I haven't used AvrUsb500 so I can't comment on that.
Remember this: the entire Optiboot bootloader takes under 512 bytes of program space. It can't be doing anything *too* complex.
You could devise your own boot loading protocol if you want to. Examples could be found in existing bootloaders, plus the datasheet for the processor suggests how to do it. You could write one that loaded code via SPI or I2C. You could build in other things. Basically you just need to somehow read in the hex file which is output from the compile process, and - in an agreed format - send that to the bootloader code, which burns it into program memory.
You haven't said *why* you want to do this, but anything is possible.
Please let me give more background of my idea,
The purpose is to download an updated firmware by host PC in the future and upload to Arduino board via Serial port.
That can make arduino having live upgrade capability.
I don't know how much effort to write customized protocol. However, that means you have to change bootloader code and also write your own program in host PC to support this customized protocol.
I checked arduino-1.0\hardware\arduino\bootloaders\atmega\ATmegaBOOT_168.c. Is it using a customized protocol?
Could you give some hints?