Example runs without 5V wire

Hey Guys,

COMPLETELY new to electronics. Picked up a Sparkfun Arduino kit. I started going through the first few examples (http://www.instructables.com/id/Arduino-Expermentation-Kit-How-to-get-Started-wi/step4/8-LED-Fun-Multiple-LEDs-CIRC02/) that basically involve setting up a ground wire, 5v wire, resistor and an LED. Why is it that when I pull out the 5V wire the LED stays lit? Does it have something to do with the fact that I'm powering it via USB and not the barrell jack? I

If you notice, in the breadboard picture of that document, nothing is actually connected to the 5V rail of the breadboard...

Your LEDs are connected to GND, through a current limiting resistor. 5V is provided by the I/O pin of the Arduino.