Attiny85 wake up slowly

Hello Guys,
I wrote a little programm which sets the Attiny85 to sleep.
I have two LEDs and a Button, if you push the putton long the LEDs will change. if you press it short the attiny goes to sleep.

I wake the tiny with the PCINT0 and it wakes up but i have to press the button 1sec that the attiny stays awake.

If I cklick the button it the tiny goes up but immediately goes back to sleep.
Is it possible to make the start time shorter?

#include "OneButton.h"
#include <avr/sleep.h>    // Sleep Modes
#include <avr/power.h>    // Power management

const byte LEDr = 1;  // pin 1
const byte LEDw = 4; // pin 4

OneButton button1(2, true);
int count = 1;
int flag = 0;


ISR (PCINT0_vect) 
 {
 // do something interesting here
 }


void setup() {
 pinMode(LEDr, OUTPUT);   // turn the LED on (HIGH is the voltage level)
 pinMode(LEDw, OUTPUT);
 digitalWrite(LEDr, HIGH);   // turn the LED on (HIGH is the voltage level)
 digitalWrite(LEDw, HIGH);
 delay(1000);
  // link the button 1 functions.
  button1.attachClick(click1);
  //button1.attachLongPressStop(longPressStop1);
 //button1.attachDuringLongPress(longPressStop1);
 button1.attachLongPressStart(longPressStop1);
 }


void loop() {
  // keep watching the push buttons:
  button1.tick();
  switch(count)
    {
      case 1:
        analogWrite(LEDr,255);
        digitalWrite(LEDw,LOW);
      break;
      case 2:
        digitalWrite(LEDr,LOW);
        analogWrite(LEDw,150);
      break;
      case 3:
        digitalWrite(LEDr,LOW);
        analogWrite(LEDw,255);
      break;
      
    } 
} // loop


// ----- button 1 callback functions

// This function will be called when the button1 was pressed 1 time (and no 2. button press followed).
void click1() {
  digitalWrite(LEDr,LOW);
  digitalWrite(LEDw,LOW);
  //set_sleep_mode(SLEEP_MODE_PWR_DOWN);
  //ADCSRA = 0;            // turn off ADC
  // pin change interrupt (example for D4)
  //PCMSK  |= bit (PCINT2);  // want pin D4 / pin 3
  //GIFR   |= bit (PCIF);    // clear any outstanding interrupts
  //GIMSK  |= bit (PCIE);    // enable pin change interrupts 
  //power_all_disable ();  // power off ADC, Timer 0 and 1, serial interface
  //sleep_enable();
  //sleep_cpu();
   
  //sleep_disable();
  //power_all_enable();
  GIFR   |= bit (PCIF);                   // clear any outstanding interrupts
  GIMSK |= _BV(PCIE);                     // Enable Pin Change Interrupts
  PCMSK |= _BV(PCINT2);                   // Use PB3 as interrupt pin
  ADCSRA &= ~_BV(ADEN);                   // ADC off
  set_sleep_mode(SLEEP_MODE_PWR_DOWN);    // replaces above statement

  sleep_enable();                         // Sets the Sleep Enable bit in the MCUCR Register (SE BIT)
  sei();                                  // Enable interrupts
  sleep_cpu();                            // sleep

  cli();                                  // Disable interrupts
  PCMSK &= ~_BV(PCINT2);                  // Turn off PB3 as interrupt pin
  sleep_disable();                        // Clear SE bit
  ADCSRA |= _BV(ADEN);                    // ADC on

  sei();                                  // Enable interrupts
  
  flag=1;
} // click1



// This function will be called once, when the button1 is released after beeing pressed for a long time.
void longPressStop1() {
 if(flag==1){
  flag=0; 
 }
 else{
  if(count<3){
     count++;
  }
  else{
    count=1;
  }
 }
} // longPressStop1

Is it possible that the switch is bouncing and therefore the tiny doesn’t get out of the wake up procedure?

Does this issue reproduce without the abomination that is the OneButton library?

It's something in your code leading to the issue (though bounce could be a contributing factor), the classic avrs wake from sleep fast (edit: not going to make quantitative claim here, as I dont know what he set fuses to)

My money is on a combination of switch bounce and how OneButton reacts to it. I dont use OneButton, so not sure how it is implemented internally, but I dont see anywhere else this behavior could be coming from