I'm having a hard time figuring out what you are doing here.
Is the battery powering something like a charge pump, or are you using a 5V generator to charge the battery?
I would suggest removing the connection to A0 for the time being and see if that helps. If you have a voltmeter, you may want to disconnect the arduino and the common ground (blue wire), and then check the voltage of the purple INA219 Vcc pin relative to the black battery GND pin. If there is some sort of current leakage from the battery, through the INA219 from V+/V- to VCC, then you could be feeding an unknown voltage to the arduino 5V bus through the arduino 5V pin.
Let us know if you can measure this with a voltmeter and what the value is.