Here is the first thing I'm working on:
http://www.oomlout.com/oom.php/products/ardx/circ-01
What is the purpose of the resistor in the setup? I see that it goes Pin 13 -> LED -> Resistor -> Ground.
Also, there is a wire between 5V on the Arduino board and the + on the bread board. However, there is nothing else connected to the + and with or without this wire the light still works. What is the function of this?
When you feed a LED a voltage over its forward voltage it will attempt to consume an unlimited amount of current. The resistor is there to limit that current to prevent the LED (or the Arduino pin!) from burning out. FWIW, typical 5mm LEDs will have a forward voltage around 2V and 20ma of current.
http://www.kpsec.freeuk.com/components/led.htm
With respect to the circuit diagram, you are correct; the 5V feeder to the breadboard is not required.