Attiny85 is unable to sleep with CR2032. Help !

Hello, I have very simple circuit with Attiny85 powered by a CR2032 battery.
The Attiny85 is interrupted by a switch (47k pull-down) and turns on 2 LEDs for awhile and goes to deep sleep again.
This works very well with 3.3V from Arduino board, however when using CR2032, after about 20 times turning on, the Attiny85 does not go to sleep again (the LEDs turn on forever)

I don't know why and how to solve this problem.

Here is the code:

//Sleep Library to interrupt MCU
#include <avr/sleep.h>

#define INTERRUPT PCINT2 //Interrupt pin

//Define for Interrupt
#define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit))
#define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit))

void setup(){
  pinMode(INTERRUPT,INPUT); //Setup interrupt pin
  pinMode(4,OUTPUT); //LED pin
  sbi(GIMSK,PCIE); // Turn on Pin Change interrupt
  sbi(PCMSK,INTERRUPT); // Which pins are affected by the interrupt
}

void loop(){
  digitalWrite(4,HIGH);delay(300);
  digitalWrite(4,LOW);delay(300);
  system_sleep(); //MCU Sleep
}

void system_sleep() {
  cbi(ADCSRA,ADEN); // Switch Analog to Digital converter OFF
  set_sleep_mode(SLEEP_MODE_PWR_DOWN); // Set sleep mode
  sleep_mode(); // System sleeps here
  sbi(ADCSRA,ADEN);  // Switch Analog to Digital converter ON
}

I guess it is because the battery gets weak and can not support attiny85 to sleep ? or something wrong with the code ?

Thanks

What's happening to the supply voltage when this problem happens?

The supply voltage is still as normal 2.45 V (with 2 LEDs) and 2.95 V when in sleep.
It remains same when this problem happens.

It seems like the Attiny85 "hangs" and if I reset it by un-plug and plug the power, it works again ...
...and after few more trying times, it hangs again...

I suspect your battery is questionable. It would also seem you have a rather large load on the batteries to reduce the terminal voltage by 20% try a larer cell as a test. That it works on the Arduino and not from batteries indicates your battery isn't capable of the load or it's no good... since the 3V3 port on the Arduino is limited in it's capacity to supply current (< 50mA). A quick suggestion would be to find some high output LED's I have some 20 candela LED's that do a fantastic job @2 mA. High efficiency red led's are fairly common and at that voltage red is all you can really use as the blues, greens and white are all 2V9 to 3V5 LED's.

Doc