Built-in LED keeps flashing when USB connected on Uno

I just received my starter's kit and started playing around with it. Everything's been working fine as far as I can tell by first uploading the blink sketch (changing delay time to see the effects etc...). But I'm wondering why the built-in LED (wired to pin13) just keeps flashing all the time when I first connected USB cable to it. I have rev2 and rev3 (from the kit) and they both have same behavior. Is it normal?

The sketch is stored in Flash ROM, and it means that the sketch is not erased at shutdown. After connecting the power supply or USB, the last uploaded sketch is running.