Thanks. I gave that a shot, and was still having trouble. Regardless of how I get the bootloader and the app (usbserial in this case) uploaded to the board, it always fails on trying to flash the board via the Arduino IDE. The error I get is:
avrdude: Send: 0 [30] [20]
avrdude: Send: 0 [30] [20]
avrdude: Send: 0 [30] [20]
avrdude: ser_recv(): programmer is not responding
avrdude: stk500_recv(): programmer is not responding
avrdude done. Thank you.
Here is a way to combine the app.hex and the bootloader.hex files together to make a combined, single hex you can upload (Thanks to Axel Wachtler for this tip!):
(sed '$d' app.hex; cat boot.hex) > app_boot.hex
This also works, but the order is important. -D in the second command prevents the flash erase.
avrdude -U app.hex; avrdude -D -U boot.hex;
My bootloader (arduino-usbdfu.hex) seems to work fine. Compiling arduino-usbserial.hex completes without errors and I can flash it to the 8U2 with no problems, but does not work when trying to upload to the main board via the Arduino IDE. I believe this is due to my avr-gcc version, which is 4.5.1. I'm going to try 4.3.3 to see if I can get Arduino-usbserial.hex to work correcly.
I can use my own usbdfu.hex, and the original Arduino-usbserial.hex, and the 8U2 will flash the main board no probs. When I compile the same Arduino-usbserial.hex and flash the 8U2, it fails. It's got to be my compiler version, as far as I can tell.
Thanks for the response! I'll post back when I have a solution.