The blink program definitely isn't running at the moment.
Then some program is running. To stop a program hold down the reset button and keep it down. I bet it doesn't happen then.
This action is perfectly normal if you are using the state of an unconnected pin to drive the LED. It's called pick up and beginners are constantly amazed by it.