See the datasheet section "Clock Startup Sequence"; startup time is 65ms PLUS some number of clock ticks (16k in the default config. It says "typical" reset times will be 69ms.)
You could change the fuse settings but you need to be aware of the time it takes the oscillator to spool up.
Crystals are the slowest to start. If you use a resonator, the startup delay might be decreasable. If you use the internal clock or an external oscillator, you can decrease even further. Some of the delay is supposedly to make sure that the power supply reaches full voltage, so if you're using the brownout detector, have a stable supply, and know that you're using reset on a system that is already fully powered, you should be in pretty good shape.