Atmega 328P power source supply problem

The button batteries measure 4.3V when not attached.

They measure 3.2V when the program is running. I'm guessing this is the problem? I'm trying to find a small power source, the whole circuit is only a few milimeters larger than the Atmega chip, so I dont really want to use 3 AAA batteries.