Thanks for all the suggestions and ideas. These will all help when I make future projects. What I had been doing was using 3 1.5v button cells (LR932) from inside a 12v battery. I made a second stack and attached in parallel for a total of 6 batteries running at 4.5 volts. It is still very small (< 2mm1mm1mm) and now when the LED is running the voltage only drops down to 3.9V, which has stopped all flickering. It seems to work well for a small project like this.
I attached a pic of the whole thing for size comparison to the ATMEGA 328p chip
