I finally got around to writing up my notes on what happens from the moment power is first applied to Arduino to when setup() and loop() are executed. It's helped me debug my current project; maybe it can help someone else. Warning: this is long and detailed.
Nice writeup but I think some details are slightly wrong:
Harvard architecture is not about having different kinds of memory. It is also not about having separate address spaces. The point is to have separate IO / memory buses.
Fuses, EEPROM and Flash rely on the same technology however fuses are NOT stored in EEPROM.
By default Arduinos are flashed with a bootloader and blink code.
The AVRISPmkII does not support JTAG thus its debugging capabilites are restricted to read memory.
You did not mention that you can set SCK 0.5 for AVRISPmkII and thus upload a lot faster than any bootloader. This is THE reason for not using a bootloader.
A hardware reset is NOT equivalent to power cycling as it does not erase the SRAM. The SRAM gets erased in software. Thus power cycling usually alters SRAM but reset does not. However it is true that almost no program will notice the difference unless it explicitly tests for it.
I'm currently using an AVR ONE! with JTAG on the EtherMega 2560 with AVR Studio 5.1 to debug some code. I've had success with breakpoints, etc. Getting the squid cable wired up to the correct pins and getting AVR Studio to co-operate has been an adventure, albeit ultimately a successful one. I'll be writing about that process in the future.