Uploading sketch from IDE works but .hex from avrdude with same options doesn't

Hi everyone
I'm trying to flash a hex file to an Arduino Nano with an Atmega168P-AU (chinese).
I can successfully flash the Blink sketch with the command (from verbose upload log):

/home/phil/arduino/arduino-1.8.5/hardware/tools/avr/bin/avrdude -C/home/phil/arduino/arduino-1.8.5/hardware/tools/avr/etc/avrdude.conf -v -patmega168 -carduino -P/dev/ttyUSB0 -b19200 -D -Uflash:w:/tmp/arduino_build_749891/Blink.ino.hex:i

but when trying to flash my own .hex file with the same command just replacing the filename, the writing skipps from about 70% to 100% immediately and then the verification fails. even if i add -n flag for no actual writes to the MCU .. so there must be a problem with me using avrdude..

my .hex file contains a page (i guess .. at least it's a lot) of 0xFFs .. and I've read that there's a problem with that .. ?

this is my output:

avrdude: Version 6.3, compiled on Jan 17 2017 at 11:00:16
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2014 Joerg Wunsch

         System wide configuration file is "/home/phil/arduino/arduino-1.8.5/hardware/tools/avr/etc/avrdude.conf"
         User configuration file is "/home/phil/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/ttyUSB0
         Using Programmer              : arduino
         Overriding Baud Rate          : 19200
         AVR Part                      : ATmega168
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65    20     4    0 no        512    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     16384  128    128  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino
         Hardware Version: 2
         Firmware Version: 1.16
         Vtarget         : 0.0 V
         Varef           : 0.0 V
         Oscillator      : Off
         SCK period      : 0.1 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.01s

avrdude: Device signature = 0x1e9406 (probably m168)
avrdude: safemode: hfuse reads as 0
avrdude: safemode: efuse reads as 0
avrdude: reading input file "/tmp/arduino_build_749891/gcton64.hex"
avrdude: writing flash (16050 bytes):

Writing | ################################################## | 100% 5.13s

avrdude: 16050 bytes of flash written
avrdude: verifying flash memory against /tmp/arduino_build_749891/gcton64.hex:
avrdude: load data flash data from input file /tmp/arduino_build_749891/gcton64.hex:
avrdude: input file /tmp/arduino_build_749891/gcton64.hex contains 16050 bytes
avrdude: reading on-chip flash data:

Reading | ################################################## | 100% 4.60s

avrdude: verifying ...
avrdude: verification error, first mismatch at byte 0x3806
         0x51 != 0x43
avrdude: verification error; content mismatch

avrdude: safemode: hfuse reads as 0
avrdude: safemode: efuse reads as 0
avrdude: safemode: Fuses OK (E:00, H:00, L:00)

avrdude done.  Thank you.

something else interesting: when i try reading the flash mem from this interactive terminal mode, it sais avrdude: stk500_recv(): programmer is not responding
this is how I connect to terminal mode:

sudo /home/phil/arduino/arduino-1.8.5/hardware/tools/avr/bin/avrdude -C/home/phil/arduino/arduino-1.8.5/hardware/tools/avr/etc/avrdude.conf -v -patmega168 -carduino -P/dev/ttyUSB0 -b19200 -t

Any Ideas?
cheers! Phil

when i try reading the flash mem from this interactive terminal mode

Don't forget that the bootloader times out. It's virtually impossible to use interactive mode on an Optiboot-based Arduino, because the timeout is only one second. The older bootloaders might give you about 10 seconds, which still isn't much.

Can you successfully upload the Blink .hex file from your command line? If so, it's likely that the problem lies in the 0xFFs, as you mentioned. Avrdude "optimizes" 0xFF pages ("I erased the chip, so I don't need to actually upload the page!"), which doesn't work because the bootloader doesn't actually do the erase. Long runs of 0xFF in PROGMEM cause upload error - Bugs & Suggestions - Arduino Forum I don't know of any fix for this. (actually, I thought newer versions of avrdude were supposed to fix it as part of their greater awareness of what pages were and weren't present in a .hex file, but I guess not.

I thought newer versions of avrdude were supposed to fix it

Um. I did some poking around, and it DOES look like the failing to upload pages of 0xFF was removed as part of the Memory Tagging Patch
Try turning on very verbose uploading (-v -v -v -v) to see if it's actually doing it, and whether it says anything informative. (I supposed it could truncate "ends of files containing only 0xFF" elsewhere in the code, but that seems wrong..)

westfw:
Um. I did some poking around, and it DOES look like the failing to upload pages of 0xFF was removed as part of the Memory Tagging Patch
Try turning on very verbose uploading (-v -v -v -v) to see if it's actually doing it, and whether it says anything informative. (I supposed it could truncate "ends of files containing only 0xFF" elsewhere in the code, but that seems wrong..)

If I remember correctly, flashing Blink from command line worked but I'll try again.
I'll definetly try the ultra verbose output and check for a newer version of avrdude I'm using the one that came with my Arduino IDE I installed a while ago. Thanks for the informative answer!
Can it be that the file I want to flash doesn't fit next to the bootloader? Is the bootloader located in the flash memory?

My file is 16050 bytes and flash has 16384 ... now I don't know what bootloader is present and how big it is.

westfw:
Don't forget that the bootloader times out. It's virtually impossible to use interactive mode on an Optiboot-based Arduino, because the timeout is only one second. The older bootloaders might give you about 10 seconds, which still isn't much.

Can you successfully upload the Blink .hex file from your command line? If so, it's likely that the problem lies in the 0xFFs, as you mentioned. Avrdude "optimizes" 0xFF pages ("I erased the chip, so I don't need to actually upload the page!"), which doesn't work because the bootloader doesn't actually do the erase. Long runs of 0xFF in PROGMEM cause upload error - Bugs & Suggestions - Arduino Forum I don't know of any fix for this. (actually, I thought newer versions of avrdude were supposed to fix it as part of their greater awareness of what pages were and weren't present in a .hex file, but I guess not.

flashing the Blink.ino.hex from commandline works fine.
Trying to flash my hex file i get the following extra verbose output log

And like you suggested it's skipping a lot of writes then fails to verify.
any idea how I can enforce those writes?

funnily enough I can't spot the 0xFF myself in the file: http://www.raphnet.net/electronique/gc_to_n64/gc_to_n64-2.1.hex

funnily enough I can't spot the 0xFF myself in the file: http://www.raphnet.net/electronique/gc_to_n64/gc_to_n64-2.1.hex

Ah hah!
It looks like the .hex file there includes a bootloader. There are two problems with this:

  1. first, it causes that big gap i the area actually programmed by avrdude, that is also not erased. This should actually be OK. Those pages are legitimately "not used", and if I had looked more carefully at your original post I would have noticed that the complaint was NOT have a 0xFF mismatch:
avrdude: verification error, first mismatch at byte 0x3806

0x51 != 0x43

  1. Second, that means the .hex file you're trying to program is meant to be programmed directly, with a device programmer, rather than being uploaded via the arduino bootloader. The bootloader in the .hex file would attempt to overwrite the arduino bootloader (which for the m168, starts at 0x3800 - right near where your error occurs.) This will either not work at all, or corrupt the arduino bootloader and essentially "brick" your system.
    If you don't need the bootloader, you can remove it pretty easily from the .hex file - it's start is obvious, and you should be able to just delete the lines between the break and the "end" marker:
[tt]:201CE000200004002000020000000200000000000000010000000100000001000000010098
:201D00000000010000000100000001000000010000000100000001000000010000000100BB
:201D20000000010000000100000002000000020000000300810003007F0003008100030010
:121D40007F00000000004743324E36347638322E31005F
:0437FC0012345678B5
[color=red]:203800000C94341C0C94431D0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C6A
:203820000C94511C0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C0C94511C20
 : Delete the part in red...
:203E60000501811115C08FEFC81AD80AE80AF80A91EFC91699E4D90692E0E906F10408F492
:203E800097CF85B1892585B9C12CD12C760190CF299A8ECF81E0111168CF66CFFB01DC01F2
:123EA00001900D920020E1F70895F894FFCF312E300062[/color]
:0400000500003800BF
:00000001FF[/tt]

For everyone else, the relevant project is documented here: Gamecube controller to N64 adapter This debugging probable would have gone faster if that had been mentioned earlier...
(Source it available, but it's pretty complex, non-arduino source.)

awesome! the flashing worked.
I'm sorry for not mentionning the source .. I thought it'd rather be something avrdude related .. when the site was talking about "Bootloader" I just thought they ment the Arduino Bootloader bot now that know that doesn't make sense :smiley:

Thank you so much!

I will try keeping the Arduino bootloader, I just hope the project bootloader doesn't initialize anything that the Arduino one doesn't.