problem porting new bootloader with avrdude

im porting a tiny version of optiboot bootloader from small avr like m8/m328 to m128/m1284. the issue is its taking avrdude almost 2 minutes to download at a high address unlike at zero which takes only a few milliseconds. for example this 3 word test program loaded at 0x00000 programs instantly:

:020000020000FC
:06000000BD9AC598FDCF7A
:00000001FF

however the same program loaded at 0x1fe00 takes forever:

:020000021000EC
:06FE0000BD9AC598FDCF7C
:00000001FF

apparently its checking or programming all the 0xff before the code or something. im using v5.11.1 and the command line is " avrdude -p m128 -c usbasp -U flash:w:t.hex". can anyone more familiar with averdude shed some light on this? is there a command line or .conf remedy? i read the avrdude docs several times and didnt see a fix. this delay is killing me.

I used a Uno to upload a bootloader, as described here:

It took 3 seconds to program a Mega.

program a mega what? you have programmed a bootloader with that into a m128 or m2560 in less than a minute using avrdude? i have several "arduino as isp" and "avrisp" programmers but they behave the same as the usbasp. it seems to be a problem with avrdude not the programmer hardware. i might be wrong about that though.

thanks for the link but the zip file on that site contains many include files and a short c program none of which resemble a pde file or compile under my arduino. im confused about just what that project is. will this somehow function as a hardware programmer for avrdude?

The Arduino Mega with the Atmega2560 chip on it.

If you unzip the download you should have the following files in a folder:

Atmega_Board_Programmer.ino
bootloader_atmega1280.h
bootloader_atmega1284.h
bootloader_atmega168.h
bootloader_atmega2560_v2.h
bootloader_atmega328.h
bootloader_atmega8.h
bootloader_lilypad328.h
gpl.txt
md5.c
md5.h

Open Atmega_Board_Programmer.ino in the IDE and it should compile without errors.

The project is nothing to do with avrdude. You connect up the wires as shown and it writes the bootloader directly from one board to the other (you control it via the serial monitor).

... you have programmed a bootloader with that into a m128 or m2560 in less than a minute ...

Three seconds, yes. But not using avrdude.

This particular sketch has the bootloaders as hex constants in the various .h files. There are instructions on the linked page for how to create such a file (so you could convert your bootloader).

Alternatively you could use this:

I am using that on a daily basis to program various chips (including the Mega2560). For example, a 19433 byte .hex file (6,902 bytes of code) uploaded in 11 seconds. The advantage of that one is you don't need to do the conversion of the hex codes into a .h file. You just copy the file onto an SD card, and plug it into the card reader (as shown in the link). Since the bootloader would be about a third of the size of the file I just uploaded, you could reckon on doing it in around 4 seconds. That's including a verify afterwards.

ahh... those are great projects and worth checking out later but might not be what im after atm. right now my requirement is specifically for bootloader development, not production or cloning arduinos. i need to cut down this edit/compile cycle time. the flexability of avrdude is hard to beat considering the number of chips i deal with. and at 2 or 3 bucks shipped on ebay a usbasp makes diy alternatives just not worth it. i was glad to see massimo and gang overcome their issues and add that to the list of supported programmers in 1.02.

speed and size of the hex is not important. my bootloaders are only 256 bytes or so. what i really need is a way to get avrdude to skip over all those unused ff locations quickly and program upper flash. any help there would be greatly appreciated.

It's been a while since I've been in the avrdude code but there was some code
at one point to try to avoid writing 0xff
It would look for a block of FFs and the remove it from the areas that need to be burned.
However it would still verify them.
It could be that the verify is what is taking all the time.

avrdude also has some really dumb sections in it with respect to reading data
depending on the interface used and the memory type accessed.
In some cases it will read blocks of 256 bytes but only use 1 byte from it. So reading
1k of data ends up reading 1000 256 bytes of data.
This is why the dragon is so slow dumping memory even though it is USB based.

Run it in verbose mode to see what is going on.
You will quickly see what it is doing.
Being able to make it better, well that is another matter.

But first you need to see what is really going on.

--- bill

john1993:
ahh... those are great projects and worth checking out later but might not be what im after atm. right now my requirement is specifically for bootloader development... any help there would be greatly appreciated.

The solution I suggested can be used to program bootloaders. In the 2 minutes you say it takes avrdude to do its stuff you could copy the file to an SD card, and program it, much faster.

Getting another programmer may not help, if you are still using avrdude. There is a programming command to erase all flash (which takes a few microseconds) which is what my programming sketch does.

as mentioned in op and youve suggested again the programmer does not matter. its the same with "arduino as avrisp", real avrisp mkII, parallel port with giveio, ponyprog, usbasp, etc. i tried them all. sd card is not an option. contacts, not to mention my fingers, would surely wear out during the thousands of downloads it takes to debug exotic boot code (no exaggeration). its also unlikely any other utility could replace avrdude considering the feature set, many of which are not optional for my applications. i simply need to make the load high time for that 3 word test program as quick as loading the same one low. ie skip ff bytes instead of doing whatever it is doing with them now.

i think, bill, you are closer to the solution. unfortunately resources are not available to consider re-compiling avrdude even if i was competent enough to understand someone elses source. if i were set up to do windows programming i might be able to throw together my own version like i did for real mode os but thats not practical in this case either.

verbose mode reveals little. erase takes no time at all and the "writing flash" progress bar simply crawls along when loading that file. avrdude is known to be a flaky buggy oddball kludge but unfortunately the closest thing to a standard and apparently nothing better is available. guess ill just have to continue limping along watching that damn program take forever to do absolutely nothing. anyway thanks for the suggestions guys and i do appreciate the replies.

john1993:
verbose mode reveals little. erase takes no time at all and the "writing flash" progress bar simply crawls along when loading that file.

"reveals little" ? Really? How many -v did you use?
My recommendation is to run with -v -v -v -v and save the output so you can see everything that is happening.
You will be able to quickly tell what is happening.
Try that and get back to us with what you see.

You should be able to tell how many blocks are being burned with what data
and then how much of the part is being read after the burn for verifying.
i.e. is it trying to burn the entire part, or it is trying to read back the full part for verifying or both?
or something else?

Once you know that you can start to look at options.
(much better than guessing)
For example,
There are options to erase the part before burning. It might be that is what enables the code that avoids burning FFs.
(I can't remember how that part of the avrdude code worked - sorry it's been quite a while since I went through the code)
Then there is the option to avoid verifying what was burned.

avrdude is known to be a flaky buggy oddball kludge

I think that is a bit harsh. I'll agree that it isn't very user friendly but
when things are hooked up correctly and the proper options are used it is quite reliable.

I don't think it is time to throw in the towel just yet.

--- bill

I went and took a look at the avrdude code.
It's starting to come back to me.
avrdude internally doesn't support multiple memory regions in the flash.
It tries to optimize the burn but it does it in a way that will
stop the burn when it reaches the highest address.

From my quick review it appears that your
example would be the worst case situation. The burn starts from the lowest
address and continues to the highest address and you end up burning and verifying all the bytes in between
with 0xff - which essentially just wastes time.

I seem to recall being involved with a discussion about this several years ago
over on the avfreaks site.
I remember there was some debate about adding in code to skip over a page being burned
if the contents are 0xff (which they are in the internal buffers for on empty regions of the flash).
But I seem to recall there was no final resolution.

If you run avrdude in really verbose mode, you will immediately be able to tell
if this is what is happening.

Another thing to keep in mind is that usbasp devices are not particularly very fast.
I think they are running at USB 1.x speeds plus the AVR is running its heart out just trying
to manage the USB. There simply isn't many cycles left over to run the ISP interface
for the burn.

Something like a Teensy board running as an ISP programmer would be much faster
because it has native USB support.

--- bill

I thought avrdude 5.11 was supposed to contain the "tagged memory" support that only programmed and verified the pages that were actually in the input file. Have you run it with the "-v -v -v -v" switches (verbose mode) to see what it is actually doing?

Maybe it does. I only had a copy of 5.10 laying around.

(the OP was running 5.11)
Is USBASP one of the programmers with "native" support for images beyond 128kbytes, or is it one of the programmers that reverts to sending raw ISP commands for every byte read/written?

thanks for the additional info. im gettting a better picture of the situation now. you might be right about usbasp throughput because in tests it takes the same to burn a 32k flash image with usb1 port as with usb2. anyway this is not an issue here because it was still only a few seconds. more than adequate and not likely related to my current problem anyway. i do wish i had more abilty to play with avrdude code. i was aware of multiple -q but only single -v so that will be my next step. feel free to try those posted 3 word test files to see what i mean. thanks again for the input.

well i tried more verbose output (-v -v -v -v) but no additional useful info results. more lines dealing with fuse safe mode but still instant erase and the writing flash progress bar creeps along when loading high. milliseconds when loading the same program low. no hint of what its doing during this time. then more safe mode comments. no time is spent before and after the progress bar and no info provided during. being an "ill behaved" cmd line app piping (">filename") does not work so i dont know what can be done to capture output for you. you can try the files in post #1 to see for yourself.

also worth mentioning at this point is i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write. another great feature of avrdude possibly related to the current problem. if you are right about it doing an unrequested verify of ff bytes that would slow things down. any more insights or suggestions?

john1993:
well i tried more verbose output (-v -v -v -v) but no additional useful info results.

This seems very odd. verbose mode spits out all kinds of stuff to indicate what it is doing.

any more insights or suggestions?

Post the verbose output here as a file attachment and so we can see what it is doing.

--- bill

as i mentioned the '>filename' method dont work. is there another way to capture output? you can also try the 2 files i put in post #1 to see for yourself. any help here would be appreciated.

john1993:
as i mentioned the '>filename' method dont work. is there another way to capture output? you can also try the 2 files i put in post #1 to see for yourself. any help here would be appreciated.

Redirection doesn't work? Really? I bet it does.

avrdude is a typical well behaved commandline tool that uses both stdout and stderr

You have to make sure to redirect everything not just stdout

 command >output.txt 2>&1

Give that a try.

--- bill

you would have won that bet. can i ask what the "2>&1" means? it didnt work w/o that and thanks for showing me that trick.

anyway heres the output file. a lot of info instantly scrolled off the screen before but i still see nothing more after the erase. does this tell us anything new?

hi.txt (6.17 KB)

john1993:
you would have won that bet. can i ask what the "2>&1" means? it didnt work w/o that and thanks for showing me that trick.

2>&1 means redirect standard i/o output stream #2 to output stream #1

This is part of "standard i/o" (stdio) which is what all unix based programs use.
The numbers represent indexes in to the file structure array.

#1 is stdout (normal output)
#2 is stderr (for errors)

This is very useful when applications use both stdout and stderr
as it allows redirecting errors separately.

This has been part of unix from the beginning for more than 40 years and unix
has been the driving and influential force for much (if not most) of what has been done in computing
over the past several decades.

Microsoft, coming to the game very late in operating systems, emulated this in
their operating system and shells starting in mid/late 80's as part of a
government requirement for POSIX compliance.

That said since "Windows" which is now using the NT kernel is based on the
VMS OS architecture vs Unix OS, some POSIX type things just don't work correctly
on Windows.

Windows makes things worse by using a monolithic approach to applications that
are almost always GUI based. As a result, many useful command line tools
and capabilities are non existent in the windows world. Like a decent scripting shell.

--- bill