I believe the fuses be set for an effective start up time of 4.1mS when 5V power is not removed.
See table 9-2 that I referenced earlier.
I made a remote control with a Promini that wakes up from power-down sleep mode after every key press. The key press causes an interrupt, the processor wakes up, scans the 4x4 keypad, and never misses a key press - I think if it was taking 65mS the presses would be missed as they would debounce themselves in that time.