Uploading sketch from IDE works but .hex from avrdude with same options doesn't

Using Arduino Avrdude, stk500, Bootloader issues
6

funnily enough I can't spot the 0xFF myself in the file: http://www.raphnet.net/electronique/gc_to_n64/gc_to_n64-2.1.hex

Ah hah!
It looks like the .hex file there includes a bootloader. There are two problems with this:

  1. first, it causes that big gap i the area actually programmed by avrdude, that is also not erased. This should actually be OK. Those pages are legitimately "not used", and if I had looked more carefully at your original post I would have noticed that the complaint was NOT have a 0xFF mismatch:
avrdude: verification error, first mismatch at byte 0x3806

0x51 != 0x43

  1. Second, that means the .hex file you're trying to program is meant to be programmed directly, with a device programmer, rather than being uploaded via the arduino bootloader. The bootloader in the .hex file would attempt to overwrite the arduino bootloader (which for the m168, starts at 0x3800 - right near where your error occurs.) This will either not work at all, or corrupt the arduino bootloader and essentially "brick" your system.
    If you don't need the bootloader, you can remove it pretty easily from the .hex file - it's start is obvious, and you should be able to just delete the lines between the break and the "end" marker:
[tt]:201CE000200004002000020000000200000000000000010000000100000001000000010098
:201D00000000010000000100000001000000010000000100000001000000010000000100BB
:201D20000000010000000100000002000000020000000300810003007F0003008100030010
:121D40007F00000000004743324E36347638322E31005F
:0437FC0012345678B5
[color=red]:203800000C94341C0C94431D0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C6A
:203820000C94511C0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C20
 : Delete the part in red...
:203E60000501811115C08FEFC81AD80AE80AF80A91EFC91699E4D90692E0E906F10408F492
:203E800097CF85B1892585B9C12CD12C760190CF299A8ECF81E0111168CF66CFFB01DC01F2
:123EA00001900D920020E1F70895F894FFCF312E300062[/color]
:0400000500003800BF
:00000001FF[/tt]

For everyone else, the relevant project is documented here: Gamecube controller to N64 adapter This debugging probable would have gone faster if that had been mentioned earlier...
(Source it available, but it's pretty complex, non-arduino source.)