problem porting new bootloader with avrdude

john1993:
you would have won that bet. can i ask what the "2>&1" means? it didnt work w/o that and thanks for showing me that trick.

2>&1 means redirect standard i/o output stream #2 to output stream #1

This is part of "standard i/o" (stdio) which is what all unix based programs use.
The numbers represent indexes in to the file structure array.

#1 is stdout (normal output)
#2 is stderr (for errors)

This is very useful when applications use both stdout and stderr
as it allows redirecting errors separately.

This has been part of unix from the beginning for more than 40 years and unix
has been the driving and influential force for much (if not most) of what has been done in computing
over the past several decades.

Microsoft, coming to the game very late in operating systems, emulated this in
their operating system and shells starting in mid/late 80's as part of a
government requirement for POSIX compliance.

That said since "Windows" which is now using the NT kernel is based on the
VMS OS architecture vs Unix OS, some POSIX type things just don't work correctly
on Windows.

Windows makes things worse by using a monolithic approach to applications that
are almost always GUI based. As a result, many useful command line tools
and capabilities are non existent in the windows world. Like a decent scripting shell.

--- bill

john1993:
a lot of info instantly scrolled off the screen before but i still see nothing more after the erase. does this tell us anything new?

There should be no output from avrdude going to the screen. Everything should be re-directed to the file.
Are you saying some output is still going to the screen?

--- bill

thanks for that description. even though unix was my first operating system and im stuck in a windows world i actually spend 99% of my time in real mode dos. so completely unaware of those details. and no, there is no output when redirecting. i was referring to what was visible before that trick.

so do you see anything in my output file that might give a hint about the slow burn?

I had to go look at the avrdude code to see why there was not more output.
Turns out that lots of debug information is actually done by the stk500v2 protocol code
rather than the upper level code in avr.c
The USBASP code doesn't have much diag output.

From the log it does appear that avrdude is writing the entire part vs just the blocks you have.
Which explains why it takes longer than when the code is down low.
But it's not conclusive since there isn't much else other than a reported size in the log.

One thing that looks odd in your output is this:

         Using Port                    : lpt1

When I use usbasp I see:

         Using Port                    : usb

Not sure why usbasp would report using the lpt port.
What is the full commandline that you are using?

In my playing with usbasp in the past, I did notice that overall the usbasp device I have is
slower than my AVR dragon at programming.

The key to speeding up burning is to burn less.
This could be handled by optimizing some code, which probably could significantly
speed things up.
There is a bit of code inside the usbasp code
that is currently commented out that will skip over 0xff.
A similar patch could be put into avrdude.

Some other things that might not require code changes:

  • Set the option to avrdude to turn off the verifying the data when burning.
  • Change the fuses to make the boot block as large as possible so the bootloader
    can live at a lower address.

BTW, what exactly are you working on?
Isn't there already a working bootloader for the 1284?

--- bill

im using v5.11.1 and the command line is " avrdude -p m128 -c usbasp -U flash:w:t.hex".

i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write.

there is no option to lower the boot address. even if that made much difference, which i dont think it would, client requires it to be in the top 256 bytes.

i developed this loader a couple years ago for smaller avr chips but now its needed for m128 and m1284. it works with any terminal program (no avrstudio or avrdude) because it accepts hex files directly instead of all that stk/109 baloney. also incorporates a bios which if im not mistaken is unique among bootloaders. and generally much faster than most isp programming.

i wonder why usbasp code to skip ff was deleted. is this the root of the problem? hard to believe since other programmers do it too. why cant avrdude skip unused locations? sounds like a bug to me. unfortunately im not in a position to modify usbasp or avrdude compiler code. only assembler atm. i see at least three others in recent posts in this forum similarly inconvenienced. and they are not involved in thousands of downloads like me. any other suggestions?

john1993:
im using v5.11.1 and the command line is " avrdude -p m128 -c usbasp -U flash:w:t.hex".

ok. So there is no -P option. I use -Pusb but it turns out that for usbasp programmer type the port
is ignored so that explains the port "lpt1" stuff.

i never turn on the verify you referred to. in my experience if theres no problem the first time then it works the next million times so no need to verify. specially since verify takes several times longer than write.

I keep hearing (reading) you say this.
But I'm not understanding what you mean because it doesn't make sense to me.
The -V option disables verification of the image after the burn. By default the burn is verified.
So it isn't something your turn on it is something you can turn off.
Using -V speeds things up because avrdude isn't reading the flash after the burn to verify it against the image.

there is no option to lower the boot address. even if that made much difference, which i dont think it would, client requires it to be in the top 256 bytes.

What I meant by this is that if you change the fuses to make the bootloader section larger,
the bootloader will start at lower address. With the largest section it will start at 0x3800
vs 0x3F00
By starting at lower address, your code will end at a lower address. That means from 0 to the
end of your bootloader code will be smaller and that will speed things up (a little).

Where the code lives or the size of the bootloader block isn't that important for debugging.

i developed this loader a couple years ago for smaller avr chips but now its needed for m128 and m1284. it works with any terminal program (no avrstudio or avrdude) because it accepts hex files directly instead of all that stk/109 baloney. also incorporates a bios which if im not mistaken is unique among bootloaders. and generally much faster than most isp programming.

I've also seen programmers that take a simple srecord file as input. While simpler, overall it isn't
as functional or as robust as a an actual program with a more advanced protocol.
Things like setting fuses or verifying the burn are not possible with the simpler interfaces.

i wonder why usbasp code to skip ff was deleted. is this the root of the problem? hard to believe since other programmers do it too. why cant avrdude skip unused locations? sounds like a bug to me. unfortunately im not in a position to modify usbasp or avrdude compiler code. only assembler atm. i see at least three others in recent posts in this forum similarly inconvenienced. and they are not involved in thousands of downloads like me. any other suggestions?

Good questions.
My personal belief is that much of this goes back to there being too many ways to update the flash
as well as some liberties taken by certain programmers/bootloaders.
The lower level code in avrdude does not know what was done prior to it being called nor what will be done after.
Some programmers/bootloaders will not erase the part even when they are told but rather erase each page
as the page is written. With those types of programmers/bootloaders skipping the 0xffs will not work.
The fastest way to update the flash is to erase it first then do full page writes only to the pages that
have something in them that needs to be written.
While its speed typically isn't an issue,
avrdude has taken the approach that working in many environments takes priority over speed.

The issue that I see impacting you is that avrdude doesn't seem to be very smart when it comes to sparse images
Internally it allocates a buffer for the full flash size, sets it to all 0xff, then reads in the data from the file.
Then it burns the flash from 0 up to the highest address used.
For a bootloader, that means you end up burning the entire part vs just the bootloader
at the top.

UPDATE:
I've just looked at the avrdude code to look for the "smarts" that westfw refered to as being in 5.11
It appears to me that this is not in the 5.11 or 5.11.1 source I have checked out
but it does appear to be in the mainline trunk code.
avr.c was updated with this "smarts" on sept 14, 2011
Now I'm really curious and will have to look closer at the revision history to see what is going on.

--- bill

It's getting more bizarre, there seems to be two different 5.11.1 versions!
On the savanah site there is a 5.11.1 from sep16, 2011 and then there is
5.11svn-20111019 which is from October 19, 2011

While there is no SVN tag for it,
the latter code from 10-19 appears to have the tagging modifications in it
to only write the locations in flash that have data from the input file.

I don't think there has been an official release since this code was added.
I'll have to see if I can get it to build.

--- bill

woohoo!!!
(problem solved)

Ok so you really WANT/NEED the updated avrdude code.
What a difference!
The time to burn an optiboot bootloader on my m328 based board,
which is 512 bytes, using an USBASP programmer used to take about 27 seconds.
Now it takes less than 3 seconds!

I don't have any tools to build it for windows so I can't help you there
but once you do get the updated avrdude, your flash bootloader updates should drop down
to just a few seconds as it sounds like your bootloader is going to be fairly small.

You need to get some kind person to build you a windows/dos version of the 5.11svn-20111019 package
or the trunk (which is what I used)
You will need the avrdude executable and the avrdude.conf as well.

--- bill

Man does Windows blow for s/w development.
After loading tools, subversion, winzip, mingw, msys, libusb and a few others
on my XP vm. I was able to build a avrdude binary from the latest svn sources.

It is attached. Rename your existing avrdude.exe and avrdude.conf and replace
them with these.
Your uploads should be MUCH faster.

Ever thought about bringing up a linux box or VM to have
a real development environment?

--- bill

avrdude.zip (423 KB)

Ah. That explains the OTHER problem I was having with 5.11 that I thought should have been fixed by the memory tagging feature. The 5.11 I had (probably) didn't have it after all !

woohoo is right! considering how much time ive already spent watching that progress bar this will eventually save me hundreds of hours. literally. mucho thanks bill. and thanks other-bill for helping with many other problems. this is a great site.

i have a quick question. why is new conf needed? ive got many custom entries in my current conf file. i notice standard old conf also gives some "id-type" error. do i have to go in there and change entries for other processor? will any conf files downloaded from other sites need to be modified too?

john1993:
i have a quick question. why is new conf needed? ive got many custom entries in my current conf file. i notice standard old conf also gives some "id-type" error. do i have to go in there and change entries for other processor? will any conf files downloaded from other sites need to be modified too?

I have no idea what is going on there. I didn't even do a diff on them to see what the differences were.
When the old one got an error, I just tried the new one and it seemed to work.

--- bill

UPDATE:
It looks like the config file format has changed slightly for additional functionality.
Which means any updates/modifications you have will need to moved into and adjusted to for the new config file.

I'm curious what mods you are needing?

oh mannnnn.... im so stuffed w/tryptophan i can hardly think. hope you guys had a good one too.

most recently i had to change the signature m1284 from 05 to 06 because of the games atmel likes to play with part number (1284 vs 1284p etc). theres a couple dozen hacks like that and a few to accommodate some of my home grown isp programmer designs and stuff like that. i thought it was something to do with changes you made to the exe but now i see its just the d&d gang up to their practical jokes again. im sure i can figure things out by comparing the two files.

btw in reference to your comment about moving my prototype code down to zero, it a nice idea and i wish. ok for tiny family but theyre so "tiny" that aint even needed. unfortunately on mega chips the booters got to be in rww section which starts up high.

anyway thanks again for a great boost in productivity. i already got more done in one day than previous two weeks.

john1993:
oh mannnnn.... im so stuffed w/tryptophan i can hardly think. hope you guys had a good one too.

:smiley: And not to mention the sugar from the pie(s)....

anyway thanks again for a great boost in productivity. i already got more done in one day than previous two weeks.

So I assume that the avrdude binary I built works for you?

--- bill

yes, so far so good. im going to try a few different devices and loading some really big flash self-check images later today but dont anticipate problems.

since you seem to be on top of this stuff more than most i have another quickie. the official 1284 bootloaders use uart 0 but the ones i got from tom carpenter use uart 1. there was mention here of issues with uart 0. do you know anything about this? which is most popular and closest to a standard?

john1993:
yes, so far so good. im going to try a few different devices and loading some really big flash self-check images later today but dont anticipate problems.

since you seem to be on top of this stuff more than most i have another quickie. the official 1284 bootloaders use uart 0 but the ones i got from tom carpenter use uart 1. there was mention here of issues with uart 0. do you know anything about this? which is most popular and closest to a standard?

I can answer that one. There seemed to be a die bug in the 1284p chip that would/could cause intermittent errors on input data stream to UART 0. It was discussed a lot over at the AVRfreaks web site. Users have reported a 'fix' if you wire a low pass filter (single small cap & resistor) network between the uart0 rec pin and the source of the transmitted data. Not sure if newer die versions of the chip have fixed the 'bug' or not. As such several have just modified their bootloader code to use UART 1 just to avoid the problem. As I said the avrfreak site had more detailed information on this problem.

http://www.avrfreaks.net/index.php?name=PNphpBB2&file=index

Lefty

thanks. me and the 'freaks' dont get along so well. some of those guys can get pretty nasty. i sometimes browse because there are helpful individuals hiding below the surface there but hesitate to even sign in. you wouldnt happen to have a link to that discussion?

also in your opinion is tom carpenters uart 1 arrangement becoming more of a standard because of this?

john1993:
thanks. me and the 'freaks' dont get along so well. some of those guys can get pretty nasty. i sometimes browse because there are helpful individuals hiding below the surface there but hesitate to even sign in. you wouldnt happen to have a link to that discussion?

also in your opinion is tom carpenters uart 1 arrangement becoming more of a standard because of this?

http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&t=107115&highlight=1284p

People building or using mega1284p in the arduino platform are not a huge population so can't say if using UART1 for the bootloader is a 'standard' or not, just one solution if one is building a new design, or as I said the simple cap/resistor seems to be effective to continue to use uart0 for the bootloader.

Lefty

thanks for taking time to do that. i guess my mistake was using 1284 instead of 1284p for search. your link has led to many other informative links not just freaks. some interesting stories.

for my nano-booter project i got 25pcs of 1284 not 1284p so maybe this is not a problem for those. with more ram than any other avr (no exception) this chip is probably going to become quite popular. making front end decisions for my bootloader im am very curious about pinout and other hardware issues involving arduino. there seems to be the sanguino and bobuino standards which i think use different uarts so it would be interesting to see what surfaces. i was also wondering why a different location for the led was chosen.