problem porting new bootloader with avrdude

john1993:
verbose mode reveals little. erase takes no time at all and the "writing flash" progress bar simply crawls along when loading that file.

"reveals little" ? Really? How many -v did you use?
My recommendation is to run with -v -v -v -v and save the output so you can see everything that is happening.
You will be able to quickly tell what is happening.
Try that and get back to us with what you see.

You should be able to tell how many blocks are being burned with what data
and then how much of the part is being read after the burn for verifying.
i.e. is it trying to burn the entire part, or it is trying to read back the full part for verifying or both?
or something else?

Once you know that you can start to look at options.
(much better than guessing)
For example,
There are options to erase the part before burning. It might be that is what enables the code that avoids burning FFs.
(I can't remember how that part of the avrdude code worked - sorry it's been quite a while since I went through the code)
Then there is the option to avoid verifying what was burned.

avrdude is known to be a flaky buggy oddball kludge

I think that is a bit harsh. I'll agree that it isn't very user friendly but
when things are hooked up correctly and the proper options are used it is quite reliable.

I don't think it is time to throw in the towel just yet.

--- bill

I went and took a look at the avrdude code.
It's starting to come back to me.
avrdude internally doesn't support multiple memory regions in the flash.
It tries to optimize the burn but it does it in a way that will
stop the burn when it reaches the highest address.

From my quick review it appears that your
example would be the worst case situation. The burn starts from the lowest
address and continues to the highest address and you end up burning and verifying all the bytes in between
with 0xff - which essentially just wastes time.

I seem to recall being involved with a discussion about this several years ago
over on the avfreaks site.
I remember there was some debate about adding in code to skip over a page being burned
if the contents are 0xff (which they are in the internal buffers for on empty regions of the flash).
But I seem to recall there was no final resolution.

If you run avrdude in really verbose mode, you will immediately be able to tell
if this is what is happening.

Another thing to keep in mind is that usbasp devices are not particularly very fast.
I think they are running at USB 1.x speeds plus the AVR is running its heart out just trying
to manage the USB. There simply isn't many cycles left over to run the ISP interface
for the burn.

Something like a Teensy board running as an ISP programmer would be much faster
because it has native USB support.

--- bill

I thought avrdude 5.11 was supposed to contain the "tagged memory" support that only programmed and verified the pages that were actually in the input file. Have you run it with the "-v -v -v -v" switches (verbose mode) to see what it is actually doing?

Maybe it does. I only had a copy of 5.10 laying around.

(the OP was running 5.11)
Is USBASP one of the programmers with "native" support for images beyond 128kbytes, or is it one of the programmers that reverts to sending raw ISP commands for every byte read/written?

thanks for the additional info. im gettting a better picture of the situation now. you might be right about usbasp throughput because in tests it takes the same to burn a 32k flash image with usb1 port as with usb2. anyway this is not an issue here because it was still only a few seconds. more than adequate and not likely related to my current problem anyway. i do wish i had more abilty to play with avrdude code. i was aware of multiple -q but only single -v so that will be my next step. feel free to try those posted 3 word test files to see what i mean. thanks again for the input.

well i tried more verbose output (-v -v -v -v) but no additional useful info results. more lines dealing with fuse safe mode but still instant erase and the writing flash progress bar creeps along when loading high. milliseconds when loading the same program low. no hint of what its doing during this time. then more safe mode comments. no time is spent before and after the progress bar and no info provided during. being an "ill behaved" cmd line app piping (">filename") does not work so i dont know what can be done to capture output for you. you can try the files in post #1 to see for yourself.

also worth mentioning at this point is i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write. another great feature of avrdude possibly related to the current problem. if you are right about it doing an unrequested verify of ff bytes that would slow things down. any more insights or suggestions?

john1993:
well i tried more verbose output (-v -v -v -v) but no additional useful info results.

This seems very odd. verbose mode spits out all kinds of stuff to indicate what it is doing.

any more insights or suggestions?

Post the verbose output here as a file attachment and so we can see what it is doing.

--- bill

as i mentioned the '>filename' method dont work. is there another way to capture output? you can also try the 2 files i put in post #1 to see for yourself. any help here would be appreciated.

john1993:
as i mentioned the '>filename' method dont work. is there another way to capture output? you can also try the 2 files i put in post #1 to see for yourself. any help here would be appreciated.

Redirection doesn't work? Really? I bet it does.

avrdude is a typical well behaved commandline tool that uses both stdout and stderr

You have to make sure to redirect everything not just stdout

 command >output.txt 2>&1

Give that a try.

--- bill

you would have won that bet. can i ask what the "2>&1" means? it didnt work w/o that and thanks for showing me that trick.

anyway heres the output file. a lot of info instantly scrolled off the screen before but i still see nothing more after the erase. does this tell us anything new?

hi.txt (6.17 KB)

john1993:
you would have won that bet. can i ask what the "2>&1" means? it didnt work w/o that and thanks for showing me that trick.

2>&1 means redirect standard i/o output stream #2 to output stream #1

This is part of "standard i/o" (stdio) which is what all unix based programs use.
The numbers represent indexes in to the file structure array.

#1 is stdout (normal output)
#2 is stderr (for errors)

This is very useful when applications use both stdout and stderr
as it allows redirecting errors separately.

This has been part of unix from the beginning for more than 40 years and unix
has been the driving and influential force for much (if not most) of what has been done in computing
over the past several decades.

Microsoft, coming to the game very late in operating systems, emulated this in
their operating system and shells starting in mid/late 80's as part of a
government requirement for POSIX compliance.

That said since "Windows" which is now using the NT kernel is based on the
VMS OS architecture vs Unix OS, some POSIX type things just don't work correctly
on Windows.

Windows makes things worse by using a monolithic approach to applications that
are almost always GUI based. As a result, many useful command line tools
and capabilities are non existent in the windows world. Like a decent scripting shell.

--- bill

john1993:
a lot of info instantly scrolled off the screen before but i still see nothing more after the erase. does this tell us anything new?

There should be no output from avrdude going to the screen. Everything should be re-directed to the file.
Are you saying some output is still going to the screen?

--- bill

thanks for that description. even though unix was my first operating system and im stuck in a windows world i actually spend 99% of my time in real mode dos. so completely unaware of those details. and no, there is no output when redirecting. i was referring to what was visible before that trick.

so do you see anything in my output file that might give a hint about the slow burn?

I had to go look at the avrdude code to see why there was not more output.
Turns out that lots of debug information is actually done by the stk500v2 protocol code
rather than the upper level code in avr.c
The USBASP code doesn't have much diag output.

From the log it does appear that avrdude is writing the entire part vs just the blocks you have.
Which explains why it takes longer than when the code is down low.
But it's not conclusive since there isn't much else other than a reported size in the log.

One thing that looks odd in your output is this:

         Using Port                    : lpt1

When I use usbasp I see:

         Using Port                    : usb

Not sure why usbasp would report using the lpt port.
What is the full commandline that you are using?

In my playing with usbasp in the past, I did notice that overall the usbasp device I have is
slower than my AVR dragon at programming.

The key to speeding up burning is to burn less.
This could be handled by optimizing some code, which probably could significantly
speed things up.
There is a bit of code inside the usbasp code
that is currently commented out that will skip over 0xff.
A similar patch could be put into avrdude.

Some other things that might not require code changes:

  • Set the option to avrdude to turn off the verifying the data when burning.
  • Change the fuses to make the boot block as large as possible so the bootloader
    can live at a lower address.

BTW, what exactly are you working on?
Isn't there already a working bootloader for the 1284?

--- bill

im using v5.11.1 and the command line is " avrdude -p m128 -c usbasp -U flash:w:t.hex".

i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write.

there is no option to lower the boot address. even if that made much difference, which i dont think it would, client requires it to be in the top 256 bytes.

i developed this loader a couple years ago for smaller avr chips but now its needed for m128 and m1284. it works with any terminal program (no avrstudio or avrdude) because it accepts hex files directly instead of all that stk/109 baloney. also incorporates a bios which if im not mistaken is unique among bootloaders. and generally much faster than most isp programming.

i wonder why usbasp code to skip ff was deleted. is this the root of the problem? hard to believe since other programmers do it too. why cant avrdude skip unused locations? sounds like a bug to me. unfortunately im not in a position to modify usbasp or avrdude compiler code. only assembler atm. i see at least three others in recent posts in this forum similarly inconvenienced. and they are not involved in thousands of downloads like me. any other suggestions?

john1993:
im using v5.11.1 and the command line is " avrdude -p m128 -c usbasp -U flash:w:t.hex".

ok. So there is no -P option. I use -Pusb but it turns out that for usbasp programmer type the port
is ignored so that explains the port "lpt1" stuff.

i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write.

I keep hearing (reading) you say this.
But I'm not understanding what you mean because it doesn't make sense to me.
The -V option disables verification of the image after the burn. By default the burn is verified.
So it isn't something your turn on it is something you can turn off.
Using -V speeds things up because avrdude isn't reading the flash after the burn to verify it against the image.

there is no option to lower the boot address. even if that made much difference, which i dont think it would, client requires it to be in the top 256 bytes.

What I meant by this is that if you change the fuses to make the bootloader section larger,
the bootloader will start at lower address. With the largest section it will start at 0x3800
vs 0x3F00
By starting at lower address, your code will end at a lower address. That means from 0 to the
end of your bootloader code will be smaller and that will speed things up (a little).

Where the code lives or the size of the bootloader block isn't that important for debugging.

i developed this loader a couple years ago for smaller avr chips but now its needed for m128 and m1284. it works with any terminal program (no avrstudio or avrdude) because it accepts hex files directly instead of all that stk/109 baloney. also incorporates a bios which if im not mistaken is unique among bootloaders. and generally much faster than most isp programming.

I've also seen programmers that take a simple srecord file as input. While simpler, overall it isn't
as functional or as robust as a an actual program with a more advanced protocol.
Things like setting fuses or verifying the burn are not possible with the simpler interfaces.

i wonder why usbasp code to skip ff was deleted. is this the root of the problem? hard to believe since other programmers do it too. why cant avrdude skip unused locations? sounds like a bug to me. unfortunately im not in a position to modify usbasp or avrdude compiler code. only assembler atm. i see at least three others in recent posts in this forum similarly inconvenienced. and they are not involved in thousands of downloads like me. any other suggestions?

Good questions.
My personal belief is that much of this goes back to there being too many ways to update the flash
as well as some liberties taken by certain programmers/bootloaders.
The lower level code in avrdude does not know what was done prior to it being called nor what will be done after.
Some programmers/bootloaders will not erase the part even when they are told but rather erase each page
as the page is written. With those types of programmers/bootloaders skipping the 0xffs will not work.
The fastest way to update the flash is to erase it first then do full page writes only to the pages that
have something in them that needs to be written.
While its speed typically isn't an issue,
avrdude has taken the approach that working in many environments takes priority over speed.

The issue that I see impacting you is that avrdude doesn't seem to be very smart when it comes to sparse images
Internally it allocates a buffer for the full flash size, sets it to all 0xff, then reads in the data from the file.
Then it burns the flash from 0 up to the highest address used.
For a bootloader, that means you end up burning the entire part vs just the bootloader
at the top.

UPDATE:
I've just looked at the avrdude code to look for the "smarts" that westfw refered to as being in 5.11
It appears to me that this is not in the 5.11 or 5.11.1 source I have checked out
but it does appear to be in the mainline trunk code.
avr.c was updated with this "smarts" on sept 14, 2011
Now I'm really curious and will have to look closer at the revision history to see what is going on.

--- bill

It's getting more bizarre, there seems to be two different 5.11.1 versions!
On the savanah site there is a 5.11.1 from sep16, 2011 and then there is
5.11svn-20111019 which is from October 19, 2011

While there is no SVN tag for it,
the latter code from 10-19 appears to have the tagging modifications in it
to only write the locations in flash that have data from the input file.

I don't think there has been an official release since this code was added.
I'll have to see if I can get it to build.

--- bill

woohoo!!!
(problem solved)

Ok so you really WANT/NEED the updated avrdude code.
What a difference!
The time to burn an optiboot bootloader on my m328 based board,
which is 512 bytes, using an USBASP programmer used to take about 27 seconds.
Now it takes less than 3 seconds!

I don't have any tools to build it for windows so I can't help you there
but once you do get the updated avrdude, your flash bootloader updates should drop down
to just a few seconds as it sounds like your bootloader is going to be fairly small.

You need to get some kind person to build you a windows/dos version of the 5.11svn-20111019 package
or the trunk (which is what I used)
You will need the avrdude executable and the avrdude.conf as well.

--- bill

Man does Windows blow for s/w development.
After loading tools, subversion, winzip, mingw, msys, libusb and a few others
on my XP vm. I was able to build a avrdude binary from the latest svn sources.

It is attached. Rename your existing avrdude.exe and avrdude.conf and replace
them with these.
Your uploads should be MUCH faster.

Ever thought about bringing up a linux box or VM to have
a real development environment?

--- bill

avrdude.zip (423 KB)