If you hook any battery at all to the 5V pin, you'll never ever get a proper 5V on the 5V
buss of the board.
Why not? That pin IS the 5V bus on the card. Whether that pin is supplied from the 5V regulator, the USB port, or directly from outside, the devices connected to it will not see a difference, and devices connected to IO pins will not see a difference.
I would put a 1N4001 diode from 5V (anode) to Vin (cathode), the currently used regulator NCP1117 is susceptible to damage from being reverse driven.
Zoomkat, sounds like you have a busted regulator.