Ok, I just realized I do not have an R3 but R2 Mega2560 (8u2). But I do have an Uno R3, so we are half way there...
Anyhow, I uploaded the 8/16u2 usbserial hex files successfully to an Uno R2(8u2) and R3(16u2) with AVRDUDE and an USBasp.
(8u2)
C:\WinAVR-20100110\bin>avrdude -p at90usb82 -F -P usb -c usbasp -U flash:w:Ardui
no-usbserial-uno.hex -U lfuse:w:0xFF:m -U hfuse:w:0xD9:m -U efuse:w:0xF4:m -U lo
ck:w:0x0F:m
(16u2)
C:\WinAVR-20100110\bin>avrdude -p at90usb[b]162[/b] -F -P usb -c usbasp -U flash:w:[b]Ardu
ino-usbserial-uno.hex[/b] -U lfuse:w:0x[b]EF[/b]:m -U hfuse:w:0xD9:m -U efuse:w:0xF4:m -U l
ock:w:0x0F:m
What I am wondering now, did you burn your Bootloader back onto the Mega2560 after you uploaded sketches with your AVRISP? Try burning the 16u2 hex with whatever gives you no errors and then through the IDE, burn the Bootloader with your AVRISP on the other header.