Exact timing of programming enable and changing fuse

I'm not sure if I can find reference somewhere of exact timing of reset signal and programming enable for atmega mcus:

All of the following numbers are in ms. Code is in ESP-IDF but I wanted to highlight delay values so I didn't put them in Code tag.

This worked. Fuse changed to programmed values:
gpio_set_level(rst,0); // mcu now enters reset
// Make sure SCK is stable. Should be, once SPI driver is initialized on bus.
vTaskDelay(pdMS_TO_TICKS(2)); // Wait 2 MCU cycles but this delay isn't that accurate.
gpio_set_level(rst,1); // mcu now exits reset
vTaskDelay(pdMS_TO_TICKS(2)); // Wait 2 MCU cycles but this delay isn't that accurate. This makes sure that a blank mcu running at 1MHz has enough time to respond to change of fuse to run at 8MHz.
gpio_set_level(rst,0); // mcu now enters reset
vTaskDelay(pdMS_TO_TICKS(20)); // Wait 20 ms before sending programming enable.
// Programming enable
send_cmd(spi,avr_isp_prg_enable,2,&readByte);

This didn't work. Fuse didn't change. (delays too long?):
gpio_set_level(rst,0); // mcu now enters reset
// Make sure SCK is stable. Should be, once SPI driver is initialized on bus.
vTaskDelay(pdMS_TO_TICKS(20)); // Wait 2 MCU cycles but this delay isn't that accurate.
gpio_set_level(rst,1); // mcu now exits reset
vTaskDelay(pdMS_TO_TICKS(200)); // Wait 2 MCU cycles but this delay isn't that accurate. This makes sure that a blank mcu running at 1MHz has enough time to respond to change of fuse to run at 8MHz.
gpio_set_level(rst,0); // mcu now enters reset
vTaskDelay(pdMS_TO_TICKS(20)); // Wait 20 ms before sending programming enable.
// Programming enable
send_cmd(spi,avr_isp_prg_enable,2,&readByte);

The reason I wanted longer delay is that these blank chips are programmed to run at 1MHz. I need them at 8MHz with external crystals. So I thought, if I program the fuses, reset, and then I can program them at 8MHz. Not working! So I thought I should add longer delay for crystal oscillator to stabilize but that caused failure to programming fuse values.

No answer to this, just more confusions. Even with reentering programming mode before each fuse write, I still don't get the fuses entirely changed at times. I wonder if anyone else has encountered this in the past. I was using a slow 128KHz SPI frequency. This seems to happen more with the 32u4 than 328p.

I'm not aware of any maximum times for the programming sequences. I've never heard of anyone having problems. For example, the optiboot makefile _isp programming sequence is (or was, anyway) done in three separate steps: erase and set fuses, upload bootloader, reset protection. This caused some problems with USB enumeration of some programmers, and were combined into a single command (presumably significantly changing the timing) without any problems. (isp targets and Arduino "burn bootloader" fail to work with AVR Dragon as programmer. · Issue #46 · Optiboot/optiboot · GitHub)

There is the clock startup time - ISP programming requires a valid clock, and the oscillator startup holds the chip in reset for up to 65ms+, so I wouldn't expect multiple reset sequences to work at very small intervals.

I'm having a hard time understanding exactly what you're doing, or how things are failing, from the descriptions and code snippets that you've been posting.

So to revisit this post, I was unable to program 32u4's fuse. Now I added chip erase and then 100ms delay before changing fuse. It works. I wonder if this is due to the fact that 32u4 comes with some stock firmware.