..have a SMALL battery to keep it alive..
Alas there is a problem there that the 5 v would become 3.8 volts (or what ever it is.)
Thinking back, to get around this problem I put a diode (same as the one after the output) between the GND pin of the 7805 and GND which would offset the drop.
Not good. You will drop the voltage even more, to 5V - 2x Diode_drop.
The best way is to put the SMALL battery (what does it mean SMALL?) at the 7805 input (btw, there are better voltage regulators today) with help of 2 diodes.