Pages: [1] 2   Go Down
Author Topic: USBasp & USBtinyISP very slow programming speed  (Read 3463 times)
0 Members and 1 Guest are viewing this topic.
Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,

Any ideas why programming an ATmega328P with the USBasp or USBtinyISP is much slower than uploading with a bootloader? For example, burning a 30kb sketch takes 238 seconds on my USBasp while serial uploading takes only about 10 seconds. USBtinyISP seems to be faster than USBasp but not nearly as fast as the bootloader.

I'd really like to use an ISP to get access to every bit of flash but the programming speed is totally killing the workflow.

Is there a magic avrdude config hidden somewhere in the Arduino IDE that controls the bitrate?

Petri
Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Ok, from the USBtinyISP page I found this: "High speed! Max clock rate is 400KHz. Write speed:1Kb/s, read speed: 2Kb/s". Apparently to enable the so called "high speed" mode I need to inject "-B 1" into the avrdude command line (default is -B 10). Don't know if it is possible with Arduino IDE.

Even with 1kb/s it would take 30 seconds to burn the sketch which is still much slower than optiboot...  smiley-confuse
Logged

Wahiawa, Hawaii
Offline Offline
God Member
*****
Karma: 36
Posts: 669
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The Optiboot bootloader is only 512 bytes long so naturally it will burn much faster than a 30K byte sketch.

Logged

Dallas, TX USA
Offline Offline
Faraday Member
**
Karma: 70
Posts: 2766
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The Optiboot bootloader is only 512 bytes long so naturally it will burn much faster than a 30K byte sketch.


Not true unless you have a newer version of avrdude than what ships with the Arduino IDE.
The issue is that the bootloader lives up high in memory, the older avrdude could not
handle holes in the image and so when you burn an image up high in flash
all the flash leading up to that is also burned and verified.
The newer avrdude can handle holes and will skip over the locations that are not part
of the image being burned.
For example, with my USBASP programmer with the older avrdude (the one that ships with the IDE),
burning a 512 byte bootloader on a atmega328 with 32k flash takes around 30 seconds.
With the new avrdude it is around 2 seconds.

Another issue with speed is that the AVRs can't be programmed fast if they
are using the internal oscillator at 1Mhz (which is the default on a blank/virgin chip).
avrdude supports the ability to set the ISP clock rate but not all the
ISP devices out there support it.
For example, many of the low cost Chinese USBASP devices are using
old firmware and don't support this option.
Most of the USBASP devices out there have a jumper to control the speed.
This was used to set the speed before the firmware allowed setting over the USB bus
by avrdude.
Most USBASP devices default to fast mode and use the jumper to  set a slow clock for ISP programming.
The issue with that is that you won't be able to program a blank AVR
unless you put in the slow clock jumper.
To ensure that blank AVRs can be programmed,
some USBASP devices are running modified firmware and are permanently
in slow mode and can never run fast. While that allows the USBASP
programmer to program new AVRS it means that re-programming chips
that are using a faster external oscillator (like on a typical UNO board)
is significantly slower than what it could be.

With the newer USBASP firmware, the
speed can be set over the USB bus by avrdude and the jumper
can be used to force the device into slow mode regardless of what avrdude
asks for.
The issue is that most of the USBASP devices I've seen out there
are not running the newer firmware.

I have no experience with USBtinyISP programmers so I'm no help there.
But with USBASP devices, to get the best speed, you need to running
in fast mode and to get the fastest bootloader uploads, you have to
use a later avrdude than what ships with the IDE.
If you are getting a warning about not being able to set the ISP clock
when using your USBASP device you are using the latest firmware.
Unfortunately if you don't get the warning you don't know if the
firmware is the official firmware or some modified version that is
stuck in slow mode.

As far as uploading speed using USBASP I upload a 31k image in 22 seconds.
That was with Arduino IDE 1.0.5 and the stock avrdude that ships with it.
It was using Linux Mint but the OS shouldn't make much, if any, difference for this.
With the serial port using the bootloader the same image takes around 11 seconds.
I'm assuming that the bootloader is faster mainly because
when using USBASP the data has to be clock in serially over ISP vs the bootloader
is able to locally load pages and write to flash in parallel.

Also the USBASP device is not only burning
the image but also implementing USB in s/w all on the same CPU.
With the bootloader, the USB and the burning are in different processors/chips.

Here is one of many threads on USBasp firmware and updating USBasp devices:
http://forum.arduino.cc/index.php?topic=149668.0

--- bill
Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

So, this confirms it. USBasp & USBtinyISP are indeed slower than the bootloader. Would Atmel AVRISP mkII be any faster?
Logged

vermont
Offline Offline
Sr. Member
****
Karma: 8
Posts: 316
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

inefficient coding and slow baud rates aside ive found the real limiting factor is flash programming delay inherent in avr chip internal hardware. ie no difference between my custom m8 based avrisp and serial 230kbaud bootloader. once you account for the overhead issues all these programming methods start to level off at 20 or so us per page.

btw i can verify the special version of avrdude bperrybap compiled for me is a huge timesaver. thanks again bill. it has saved me many hours sitting staring at that ridiculous progress bar. maybe im not doing thing right but the only drawback is incompatibility with existing config files. its been necessary to re-edit the special version for any new devices added instead of simple download of one of the official ones.

i wonder if have any of the original avrdude developers caught up with bill yet and snapped out of their funk.
Logged

Dallas, TX USA
Offline Offline
Faraday Member
**
Karma: 70
Posts: 2766
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

inefficient coding and slow baud rates aside ive found the real limiting factor is flash programming delay inherent in avr chip internal hardware. ie no difference between my custom m8 based avrisp and serial 230kbaud bootloader. once you account for the overhead issues all these programming methods start to level off at 20 or so us per page.

Some of the reduced performance is the way avrdude uses the devices and the interfaces
that talk to them.
For example, on some devices it internally emulates the internal interfaces to
other ISP devices to take advantage of some existing internal routines.
This is particularly true on some of the devices that use the USB interface.
The AVR dragon is one of them. avrdude is not using the USB bus efficiently for some operations
and will send single bytes per USB frame rather than 64 byte frames. This dramatically slows
down operations from they could be.



btw i can verify the special version of avrdude bperrybap compiled for me is a huge timesaver. thanks again bill. it has saved me many hours sitting staring at that ridiculous progress bar. maybe im not doing thing right but the only drawback is incompatibility with existing config files. its been necessary to re-edit the special version for any new devices added instead of simple download of one of the official ones.

I didn't experience this. The newer avrdude does come with its own newer config file.
The later avrdude does use a slightly different config file format but I haven't had to edit it yet
to support any of the devices I have that are supported by the Arduino s/w or the 3rd party core addons.
(But I've only used it for m328, attiny25, attiny45, and attiny85, attiny2313)


i wonder if have any of the original avrdude developers caught up with bill yet and snapped out of their funk.

Just to be clear, I didn't modify the code to make avrdude faster by knowing about sparse .hex images.
That is a stock feature of the newer avrdude code.
The real issue is that the Arduino team is dragging their feet. They are using
a version of avrdude that is like 5 years old. If anyone needs scolding it is the Arduino team
for using such an old version of the code.


So, this confirms it. USBasp & USBtinyISP are indeed slower than the bootloader. Would Atmel AVRISP mkII be any faster?

With respect to the speed of the AVRISP mkii, I don't know.
But I would guess that its performance is probably pretty good
since it is in such widespread use (vs compared the AVR dragon)
and probably more likely to get the attention of the avrdude developers.

Some of these slow downs are due to to how Avrdude uses the devices vs the devices themselves.
But much of the slow down like what you are seeing where USBASP is significantly slower is
do to mismatched firmware and the way the IDE is telling avrdude to use the device.
Fix the USBASP firmware and you will speed up your programming by 10X.
And then there is libusb. libusb and avrdude have some "playing nice together" issues.
This tends to affect some devices more than others. It is a really big issue for the AVR dragon.
Like you can't use it with the stock Makefiles for optiboot and to burn a bootloader using the IDE.
There are some simple work arounds that can be done in the optiboot makefiles or in the IDE
and there is a small fix in avrdude itself that can permantly fix it.
So far I've been unable to get the Arduino team or the avrdude maintainer to make some small
changes to make things work.
For this issue I did make a change to the avrdude code to make it work properly.
The AVR dragon is an area where I think the avrdude team is dragging their feet.
It is ironic that the AVR dragon, an official Atmel product. is an expensive design with tons of chips and yet its performance
capabilities are often lost by the way avrdude uses it.
While I like it because it can do literally "everything" and I've even used for
source level debugging Arduino code running on an Attiny85 using
the single wire interface, its current ISP programming performance
using the stock avrdude is a let down.
My cheap USBASP device is quite a bit faster at ISP programming.


I would think a Teensy would make a fast ISP programmer.
Since it has native USB support it should be able to push the data over USB
substantially faster than USBASP and USBTiny and is not limited by
an actual serial bootloader connection like those devices that use a USB type device
(FTDI or Atmel chip ) that is converting the data to a serial interface for a serial bootloader.

Guess I should try it sometime.



Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Thank you for the insightful replies!

Before I go and buy yet another ISP, it would be much appreciated if somebody who has experience with these "better ISPs" would pipe up. I'm thinking of getting an AVRISP mkII but it's a bit pricy here in Finland, so I would rather make it sure that it really is better before dishing out the money.
Logged

Dallas, TX USA
Offline Offline
Faraday Member
**
Karma: 70
Posts: 2766
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Ok so here is additional information on USBasp.
So I decided to dig deeper and really see what is going on as to why the USBasp is not faster.
Is it the USBasp device, USBasp firmware, avrdude, the IDE or some combination?
I have some answers.

The the newer USBasp firmware (since 2009) supports setting the SCK clock speed over the USB.
There is a default clock rate built into the firmware.
It is 375 kHz.
The USBasp interface supports telling the device to automatically pick a clock.
The newer avrdude uses this when connecting to the USBasp device.
The USBasp firmware doesn't really auto detect the SCK rate, instead it just
sets it the default clock rate which again is 375 kHz.

The USBasp message interface supports setting higher clock rates over the USB interface.
It supports  750 kHz and 1.5 Mhz.

So now the trick is how to get the USBasp device to run the higher clock rates.
Before we go into that, I went and looked up just how fast is the fastest SCK can be clocked.
With clocks > 12Mhz which the 16Mhz AVR is, you can't clock it faster than 6 clocks which means
no faster than 2.666Mhz.
So great I thought I'd just whip up special version of USBasp firmware that defaults
to running the SCK clock at a frequency faster than the current default of 375Khz
and right up as close to 2.666 as I can get,  assuming I should be able to get 2Mhz.
Well you can't do 2Mhz. The USBasp device uses AVR SPI hardware clocked from power of 2 divisors off
the main clock. The main clock on USBasp is 12Mhz so you can do /8 to get 1.5Mhz and then /4 to get 3Mhz
but nothing in between.

So now I thought about burning a new version of the firmware with a faster SCK at 1.5Mhz but
first starting looking at using -B options to set the clock faster, as it is a bit of pain to get
things setup to burn new firmware into the USBasp device I have.
While I was running avrdude from the commandline for my testing,
I was a bit confused at first since the -B option value is not passed directly to the USBasp
firmware. I was assuming -B10 or -B12 was passed in directly through the lowlevel USBasp messages.
It was a user error on my part. The -B option is actually a floating point value
that represents the bit clock frequency in microseconds. This is actually nice since it is consistent
on all devices regardless of how the low level messages are implemented.
To to get a frequency of 1.5Mhz on you use -B0.666
(which avrdude turns into a low level USBasp message clock value of 12)

Testing showed that the using -B0.666 is much faster than using the defaults.

I went looking to see how the IDE runs avrdude and see what -B options it is using.
Well it turns out that the IDE is not using a -B option.
However, after looking at the java code there is a way to trick it to allow setting a -B option.
What you need to do is go into your programmers.txt file and modify the entry for the USBasp programmer.
change this line:
Code:
usbasp.protocol=usbasp
to this:
Code:
usbasp.protocol=usbasp -B0.666
Alternatively you could create new programmers for fast/slow programming etc..:
Code:
usbaspSlow.name=USBaspSlow
usbaspSlow.communication=usb
usbaspSlow.protocol=usbasp -B10

usbaspFast.name=USBaspFast
usbaspFast.communication=usb
usbaspFast.protocol=usbasp -B0.666
I would have preferred to put the -B option on the .communications line
but the java code doesn't use it. It looks for "usb" on the line then hardcodes the option to "usb"
rather than just using the string.
But you can trick it by adding an additional option to the protocol field
as the java code just blindly uses whatever you put there "as is".


So by using these modified USBasp faster programmer types,
the USBasp SCK clock will be 8 times faster than the current default clock.
Before you jump up and down, that doesn't translate into an upload time that is 8 times shorter.
However it does drop a full 32k 25 second upload down to about 12 seconds.
While still not quite as fast as the Serial upload it is pretty close.
And when using the new avrdude burning a optiboot bootloader drops from
around 40 seconds down to about 0.5 seconds.

Just keep in mind that you have to have the newer USBasp firmware that
supports setting the clock over the USB message interface in order to do this.
And if you are building your own USBasp firmware image, I recommend going
in and changing the default clock rate to be 1.5Mhz instead of 375Khz.
It is a one line change in isp.c
Change this:
Code:
if (option == USBASP_ISP_SCK_AUTO)
option = USBASP_ISP_SCK_375;
to this:
Code:
if (option == USBASP_ISP_SCK_AUTO)
option = USBASP_ISP_SCK_1500;

If you make this firmware change, you won't need to modify your boards.txt file
to get the faster clock as the default SCK clock will be 1.5Mhz

--- bill


« Last Edit: July 28, 2013, 04:06:55 pm by bperrybap » Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Wow, I didn't expect anyone to dig into this so deeply. Very kind of you, bperrybap!

I'll try the -B option trick right away when I get from work!
Logged

vermont
Offline Offline
Sr. Member
****
Karma: 8
Posts: 316
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Testing showed that the using -B0.666 is much faster than using the defaults.

be aware avr can vary considerably regarding flash time. ive seen cases where one chip required 2x more (40us vs 20us) than another of the same type. also big variation from a new chip to one thats been programmed few hundred thousand times. in no case would i choose a speed that violates minimum timing spec.

if i were producing usbasp id choose the slowest rate, not fastest, as default so guaranteed to work with 1mhz chips out of the tube. then make sure latest firmware is used so hot shots can boost the rate for more productive hacking if desired.
Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I tried the -B trick but it does not work... I get this error when using Arduino IDE:

Code:
Binary sketch size: 1,084 bytes (of a 32,256 byte maximum)
/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/bin/avrdude -C/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf -v -v -v -v -patmega328p -cusbasp -B0.666 -Pusb -Uflash:w:/var/folders/d4/9j9j2dbx0b75sxlklqbgz68w0000gn/T/build4467686291424749628.tmp/Blink.cpp.hex:i

avrdude: Version 5.11, compiled on Sep  2 2011 at 18:52:52
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/Applications/Arduino.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf"
         User configuration file is "/Users/petri/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping


avrdude: Can't find programmer id "usbasp -B0.666"

I tried with different settings of B value (-B1, -B2, etc.) and tried adding a space between the B and the number.

The weirdest thing is, it works perfectly if I copy the command from the Arduino output to command line! Feels like a bug in the IDE. I'm clueless...
Logged

Dallas, TX USA
Offline Offline
Faraday Member
**
Karma: 70
Posts: 2766
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

What version of the IDE are you using?

I was using 1.0.5 and didn't look at any of the older versions.
I was also on linux, but since it's java code theoretically it shouldn't matter

--- bill
Logged

Dallas, TX USA
Offline Offline
Faraday Member
**
Karma: 70
Posts: 2766
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I tried with different settings of B value (-B1, -B2, etc.) and tried adding a space between the B and the number.

The weirdest thing is, it works perfectly if I copy the command from the Arduino output to command line! Feels like a bug in the IDE. I'm clueless...

OOPs... Sorry guys, in my testing I had a wrapper script to catch the avrdude execution.
It would print out the command then call the real command.
The side effect of this is that the white space between arguments gets re-interpreted.
If I go back to not having the wrapper script it fails the same way.
For those interested, what I did was go in and rename avrdude to avrdude.cmd
then use this wrapper script called "avrdude":

Code:
#!/bin/sh
avrdude_cmd=${0%/*}/avrdude.cmd
echo AVRDUDE $@
$avrdude_cmd $@

You could still use this assuming you are on a *nix OS to make it work,
although you might want to remove the echo command.

--- bill
Logged

Finland
Offline Offline
Jr. Member
**
Karma: 1
Posts: 93
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Ah, ok, that explains it! No worries smiley

Hmm, maybe we should post a feature request about the possibility of having custom flags passed to avrdude. Would fit rather well in programmers.txt and would be a rather nice general improvement. Easy to implement hopefully too.
Logged

Pages: [1] 2   Go Up
Jump to: