Arduino Forum

Using Arduino => Microcontrollers => Topic started by: Geoffrey_Brown on Mar 15, 2019, 01:01 am

Title: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: Geoffrey_Brown on Mar 15, 2019, 01:01 am
EQUIPMENT
I have a Mega 2560 and Windows 10 laptop that I am using to interrogate flash, fuses etc using avrdude in both terminal (-t) mode and also using the -U option (ie. -Uflash:r:flashdump.bin:r) to get stuff off the Mega 2560.

I also have a USBasp programmer that plugs into the USB port and connects to the ICSP interface.


ISSUES
Unfortunately I have a range of issues with the results I am seeing.  Have spent a couple of days on this and am getting nowhere.


I am getting very confused about some of the results I am seeing.   If I use AVRDUDE to connect using the interface wiring (-c wiring) then things partially work.   Looking at boards.conf it looks like the wiring interface (basically USB I believe) is used to upload sketches which all works fine with the Arduino IDE - and this is working fine.

If however I use:

avrdude -p m2560 -c wiring -P COM4 -C <config file C:\Program Files...> -t

to put it in interactive mode and:

  - dump flash: always get 0's (always incorrect)
  - lfuse: reports 0xff (this is correct)
  - hfuse: reports 0xd0 (this is correct)
  - efuse: reports 0xff (this is correct)
  - lock: reports 0x10 (should be 0x3f)
  - changing any fuses usually results in an error like 0x10 != 0x3f and reports verification failed.

If I change the avrdude to use -c stk500 it rarely works at all and typically fails to get into interactive mode and has the following errors:

  - multiple stk500v2_ReceiveMessage() errors.

Now if I use the USBasp interface:

avrdude -p m2560 -c usbasp -C <config file> -t

I get the following results:

  - if I use dump flash 0 500 or dump flash 45000 500 I get data this corresponds to my uploaded sketch so all good.
  - if I use avrdude to dump flash (avrdude -p m2560 -c usbasp -C... -Uflash:r:flashdump.bin:r) I get what appears to be the bootloader included in the file created but the loaded sketch is all 0xff.   
  - no matter what format I use to dump flash (Intel = :i, Hex = :h) I always get what I would call a partial flash dump of the bootloader only.
  - looking at any of the fuses works and all values are correct and I can change their values.
  - if I look at flash at address 253952 I see what I think is the bootloader (address equivalent 0x3e000).   

THINGS I HAVE TRIED
Having tried all the tests already mentioned I have then looked at fuses in case they are causing issues.   The only fuse that appears to be relevant is the lock fuses that control access to memory for bootloader, app etc.   With the settings of 0x3f this appears to allow full access.

I am therefore confused about the following:

1) The Arduino IDE uses the USB alone to upload a sketch, verify it, then run it.   I cannot get the wiring or stk500 interface options with avrdude to even read flash let alone set fuses.   Logically this is working because the Arduino IDE is working.

2) While the USBasp interface works in interactive mode perfectly and I can change fuses & lock and read them fine when the non-terminal mode is entered and a flash dump is done is it only partially correct.

3) If I write the flash read done via USBasp and use USBasp to perform the write I end up with the bootloader being restored which I can see with the flashing LED.   My application is missing however thereby confirming the issue with the flash that was read.

I have Arduino V1.8.8 and this frustration is really spoiling what I have achieved with Arduino so far.   Any help would therefore be much appreciated.

Geoffrey, New Zealand.

Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: WattsThat on Mar 15, 2019, 02:09 am
What you fail to explain is why you're thrashing about with the usbasp. You only need it to install bootloaders on new, blank chips that aren't mounted on an Arduino board.

Burning the bootloader erases all of flash, your sketch is wiped in that process.
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: Geoffrey_Brown on Mar 15, 2019, 05:33 am
I wanted a copy of everything in flash and eeprom so that I could use a backup file to restore these areas.   In particular the EEPROM.  In the process I discovered the mess in using ICSP with Mega2560 Arduinos.  Should be a lot simpler than this and I believe there is something going wrong with the ICSP / USBasp process.

Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: Geoffrey_Brown on Mar 15, 2019, 05:43 am
The mention by WattsThat about the burning of the bootloader clears all of flash I am aware of but the strange operation of avrdude with usbasp and the mega 2560 is the focus of this thread.

Geoffrey
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: westfw on Mar 15, 2019, 07:27 am
The Arduino bootloaders all (well, Optiboot, anyway) time out relatively quickly and execute the user sketch.  This means that the "interactive" (-t) mode of avrdude is essentially useless for poking at them.

I'm not sure about your USBasp problems.  Some of the inexpensive firmware for those progammers has trouble with the parts having more than 128k of flash, I think.

Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: pert on Mar 15, 2019, 07:52 am
Here's the firmware that gives the USBasp full compatibility with your ATmega2560:
https://github.com/PeterVH/usbasp/tree/v1.06 (https://github.com/PeterVH/usbasp/tree/v1.06)
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: Geoffrey_Brown on Mar 17, 2019, 11:53 pm
With the issues with the Mega2560 that I have discovered when using it with the USBasp ICSP programmer I thought I would try to make several other people's lives easier but putting the full facts out there.   Feel free to correct me if I have things wrong in the summary below.

This issue with the Mega 2560 and the USBasp programmer connected to the ICSP port has got unfortunately three facets to it which I will go over below.

BACKGROUND
The first is that the Mega 2560 with 256KB is the only AVR micro-controller that I am aware of in common use that exceeds the 128KB program size (flash).   Most other micro-controllers are in the program size (flash) range of 8-32KB.

ISSUE 1 - NEW ISSUE CAUSED BY LARGE FLASH SIZE OF MEGA 2560
The first issue relates to the operation of the ICSP port on the Mega2560 (not the ICSP port for the 8U2 or 16U2 micro-controller for USB<->Serial comms).   The issue here is that the ICSP protocol uses two functions to set the address:

1) The Load Program Memory Page (LPMP) which takes a 16 bit address (ie. 0x0000 - 0xFFFF).

2) The Load Extended Address Byte (LEAB) which deals with addresses that exceed what 1) can handle.   This sets the part of the address higher than the 16 bits allowed in LPMP.

For most processors the LEAB is not needed as the address fits within the LPMP range.   The address of 0xFFFF corresponds to the 64KB boundary but because this is a WORD address (not BYTE address) the actual memory range LPMP handles is therefore 128KB.  A byte is 8bits and a word is 16bits.

TAKEHOME MESSAGE:
The take home message is that the Mega 2560 needs to use the Load Extended Address Byte (LEAB) function to access all of memory correctly.   Any programmer using the ICSP interface MUST implement the LEAB function correctly.   This will not be an issue for any other AVR micro-controller that has less than 128KB of flash, eeprom etc.   Most other micro-controllers do not have more than 128KB of program space (flash).

ISSUE 2 - THOMAS FISCHL FIRMWARE ISSUE
While there are several USBasp programmers made from different companies they largely operate off firmware from Thomas Fischl who looks to be a German chap.   The second problem comes to a head with firmware V1.04 (usually dated as being 28 May 2011 version).   This firmware purports to support Mega2560 extended address space but has a bug that causes another set of issues.

In summary the issues are:

a) PART A OF ISSUE 2 (HFUSE IS 0xD8)
If the hfuse of Mega2560 is set to 0xD8 then it will be possible to program the second half of the flash address space (corresponds to byte addresses from 0x20000 to 0x3FFFF).   The fuse setting for the Mega2560 typically has the bootloader located at word address 0x1F000.  This corresponds to byte address 0x3E000.   This means the normal fuse setting of hfuse=0xD8 allows the bootloader to be burnt to flash.   In fact it will also burn the bootloader at address 0x1E000 as well but it will never get run from there.   This burning at 0x1E000 is pointless but is an issue related to not handling the 0x3 part of the address correctly.

b) PART B OF ISSUE 2 (HFUSE IS 0xD9)
The other scenario you get is if the hfuse is set to 0xD9.   This will not execute the bootloader but instead immediately executes the users application.   When this fuse is set it will be possible to correctly program the lower address space (bottom 128KB) but the top 128KB will just be a copy of the lower address space -  the users program will be loaded a second time at an address that will not be used - like the situation with PART A ISSUE 2.   The users program will basically be loaded at address 0x00000 and will also be loaded at 0x20000 (start of second 128KB of flash).
A related issue here is that I purchased my USBasp from Jaycar in 2018 and despite being well after the V1.04 firmware creation date in 2011 it still had V1.02 on it.   Even though I upgraded it to the latest version (V1.04) the bug I have just documented here as issue 2 then raised its head.

TAKEHOME MESSAGE:
So even if you do the upgrade to USBasp V1.04 from Thomas Fischl you are stuffed if you are wanting to load both the bootloader AND your application into flash - or make a complete copy of flash in the Mega2560.   In my case I wanted to make a full backup of the Mega2560 and that was also not possible.   The file that is created has had no errors flagged in the process but it will not work if you use it to reload your Mega2560 and you want the bootloader AND your sketch (application) loaded back again.
The command to dump what is in flash should be (from cmd prompt):

avrdude -p m2560 -c usbasp -C <confile file> -Uflash:r:flashbackup:r

but if written back to flash with hfuse = 0xD8 the bootloader will be there but not your application (sketch):
avrdude -p m2560 -c usbasp -C <config file> -Uflash:w:flashbackup:r
You need to download a version of the firmware which properly works with the Mega2560 and this one appears to do just this:
https://github.com/PeterVH/usbasp/tree/v1.06



ISSUE 3 - USING AVRDUDE WITH -p wiring HAS TIMEOUT ISSUES
The last issue is that I thought given the Arduino IDE uses the wiring interface to program the sketches into the Arduino, and this works, could I bypass the USBasp programmer and just use the AtMega16U2 USB to Serial chip to do what I want.   After all this is how the Arduino IDE uploads my sketches.

The answer is no - well not in terminal mode.   When using avrdude to connect to the Mega2560 I ran it from a cmd prompt as follows:
avrdude -p m2560 -c wiring -C <config file location> -B 19200 -P COM4 -t

With my limited understanding I would have thought this would have worked.   Unfortunately reading the fuses was only partially successful and reading flash or eeprom was always unsuccessful:

in avrdude:
avrdude> dump flash 0 300 - FAILS
avrdude> dump eeprom 0 300 - FAILS
avrdude> dump hfuse or dump efuse etc - PARTIALLY SUCCESSFUL

When the -t switch is used with wiring because it talks via the Mega16U2 chip there is a timeout built into the firmware on this chip that largely makes using this method hopeless.   It will timeout before the user generally enters too many commands.  
You can use to wiring interface to dump the flash out:
avrdude -p m2560 -c wiring -C <config file location> -B 19200 -P COM4 -Uflash:r:flashbackup.bin:r
because there is no delay waiting for the user to enter the command but unless you know about this timeout preventing -t (interactive terminal mode) from working you are stuffed again.

TAKEHOME MESSAGE:
The wiring interface which is what the Arduino IDE defaults to will work with the Mega2560 but unless you avoid terminal mode your mileage will vary.   For the beginner in AVR & Arduino there are limitations which preclude the use of terminal mode as a second option to doing basic things like dumping flash or eeprom when using the USBasp programmer.   Your output will show all 0x00's which is crap.    You will also get the occasional stk500_v2 error.   When the USBasp programmer is connected instead the avrdude terminal mode there is no timeout issue.

Overall a painful learning experience and one that I hope may help you.
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: westfw on Mar 18, 2019, 04:47 am
Quote
When the -t switch is used with wiring because it talks via the Mega16U2 chip there is a timeout built into the firmware on this chip that largely makes using this method hopeless.
It's not avrdude, or the 16u2 firmware, but the bootloader running on the 2560 itself that has a built-in timeout.

Thank you for writing up your findings, but ... please don't "cross post" (ie put the same message in several different topics/etc.)

You might be interesting in knowing about the "monitor" program that I think is still present in the MEGA bootloader.
If you type three consecutive exclamation points ("!!!") during the time that the bootloader is running, you'll get a baby interpreter that lets you examine a number of the microcontroller bits:

https://github.com/arduino/ArduinoCore-avr/blob/master/bootloaders/stk500v2/stk500boot.c#L1271 (https://github.com/arduino/ArduinoCore-avr/blob/master/bootloaders/stk500v2/stk500boot.c#L1271)

I think this is the first time I've seen anyone other than myself try to use "advrude -t" with an arduino.   I imagine that the actual timeout period (~10s, IIRC?) is particularly annoying, since it gives you just enough time to have one or two commands "work" before it fails...

(The monitor is similarly unpopular.  It's mainly known for early-day bootloader bugs that caused any program upload with three consecutive exclamation points in a string to fail mysteriously.)
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: Geoffrey_Brown on Mar 18, 2019, 09:09 am
Thanks Westfw for clarifying what is happening with the 16u2 and mega2560.   Most interesting.

You are absolutely correct in that the timeout is very annoying and initially you think you are going mad and then after more alcohol you realize it is time based.

In regards to cross posting the reason I put the solution against several threads was because in trying to find a solution I came across all these threads (still had them as separate tabs in my browser) and they failed to address the real issues or provided workarounds that were unsatisfactory or useless in other scenarios (like mine).   These workarounds typically allowed only one half of the 256KB flash to be addressed at a time which is still useless in various situations like what I experienced.

The other issue you have here is that these old threads are still fully indexed on search engines like google and unless they are deleted they act as a source of incorrect or poor information for eternity.   The cross posting is therefore to ensure that those web pages I came across trying to find a solution are now corrected for others in the future.   You are otherwise looking for a needle in a haystack trying to find an answer.   I would have had 30+ tabs open trying to figure this issue out.

There is no other purpose or outcome intended by me other than sparing others wasting days with a similar problem.  Unfortunately, I fail to see how others can get the answers they seek amongst a sea of poor information, which is what is currently the state with all the past historic posts. If nothing else readers are aware of the issue and can recognize the issue if using other programmers which are having the same problems.

If you have a solution which provides the correct answer to the multiple past threads that have been started around this problem in the past I am all ears.   What I have done seems to me a sensible thing to do to reduce frustration and confusion and instead provide answers and clarity.
Title: Re: AVRDUDE read options (flash, fuses) not dumping all data out to file
Post by: pert on Mar 18, 2019, 10:39 am
It would be better to document your findings in the most appropriate thread, then post links in the other relevant threads to the one location where you posted the full explanation. That way, it's less likely to cause parallel discussions

I fail to see how others can get the answers they seek amongst a sea of poor information
I ran into the problem with the USBasp and ATmega2560 and I found the issue to be quite well documented at that time. That was before the fixed USBasp firmware was available, but it seems like we've done a pretty good job of getting the word out about it.

I'd say my primary reference on the subject has been this issue report:
https://github.com/arduino/Arduino/issues/388 (https://github.com/arduino/Arduino/issues/388)