We need more information.Where are you connecting this 5V power supply "directly" to your board. Is it through the power jack? If so you need about 7.5V to drive the regulator.
in my head that doesnt make sense
I can tell you that there is a 75% chance that there is anther problem preventing it from working! The other 25% would be that the wires do not have enough contact when you twisted them together. Try soldering them and see what happens!
I don't see any power supply decoupling onthat board. Try soldering a 0.1uF ceramic cap under the board across the power rails. The bread board is possibly adding a bit of capacitance to the circuit.
Are you sure that there is not some other factor in play here? Check the voltage coming from the power supply. Is it consistent?