Arduino circuit/breadboard not working at higher voltage?

Almost correct. That larger capacitor you have wired between ground and +5vdc should be changed so that it's negitive lead is still wired to ground as you presently have it, but it's positive lead is connected to your regulator's input pin, which is any of the pins in row two on your picture.

Lefty