As retrolefty pointed out, for proper LED circuits, you should put a resistor between the Digital pin on the arduino board and the long pin of the LED. This protects it and the arduino.
Those tutorials by Ladyada that I linked in the previous post are a good source for beginners. It'll teach you basic circuits and basic arduino programming.