OSCCAL, bootloaders and internal oscillators

Hi! I need some good advices here, so here it goes.

I'm the maintainer of the MightyCore, which adds Arduino support for the ATmega1284, ATmega644, ATmega324, ATmega164, ATmega32, ATmega16 and ATmega8535. As you may know they are all using Optiboot, and one can choose in the boards menu whenever you want to use an internal or an external oscillator (8MHz or 1MHz internal). This is a nice feature if your application doesn't depend on super high speed or accurate timing.

As you may know the internal oscillator isn't super accurate, and it may need to be manually calibrated. As far as I've understood the calibration bits (in the OSCCAL register) isn't "burnt in", and Atmel recommends to store this data in the EEPROM and load these into the OSCCAL register when the program starts. This is OK if you're programming using an ISP adapter, but when using a bootloader one need this calibration to be stored in the register before the program is loaded. Is this even possible?

For the MightyCore 8MHz internal oscillator option I've been using 57600 baud as upload speed. This baud rate have been working great with the ATmega32, ATmega16 and ATmega8535, but not with the rest. According to WormFood's baud rate calculator the baud rate of 57600 baud gives significant error, but the lower 38400 baud is just 0.2%. After compiling the 38400 baud bootloader, the ATmega1284/644/324/164/32 works, but the ATmega16/8535 won't sync up.

My goal is to make all microcontrollers work no matter how "off" the internal oscillator is. Is it possible to use the "new" OSCCAL values when you're uploading using UART and a bootloader? Is a baud rate of 19200 or 9600 "acceptable" when 38400 baud theoretically should work?

What is the reference ? There must be a known frequency somewhere to adjust the OSCCAL.
Have you seen : "AVR053: Calibration of the internal RC oscillator".

I have used the OSCCAL in a project for internal oscillator. I use the temperature and the battery voltage (it runs with three AA batteries) and with those I calculate the OSCCAL. But I have done that with trial and error and it works only for that ATmega chip.

When it should work no matter what, you have to know the temperature and voltage, to be able to adjut OSCCAL while running.

I already use the 38400 baud, for small projects and with internal oscillator.
I have no solution for this. There is only one good solution : use a crystal.

I've had a look at AVR053. The reference will in my case be a frequency generator. But what about the "correct" OSCCAL value? I don't want to hard code this into the bootloader either. I'll try 19200 or 9600 baud and see if all microcontrollers will accept it, even with uncalibrated oscillators. This might be the easiest option for everyone..

That will not always work. Even the 38400 baud will not always work.
Because of that, there is not safe way to start the communication.

I did some tests with the signature row : https://forum.arduino.cc/index.php?topic=392433.msg2708936#msg2708936
The factory calibrated value for OSCCAL is in the signature row. It is copied to OSCCAL during power on.
The signature row is read-only, you can not change that.
The OSCCAL is a normal register, just like many other registers.
The EEPROM is where the new OSCCAL could be stored, but the complete EEPROM could be used by the sketch.

It is possible to adjust OSCCAL for the exact baudrate, so the 19200 or 9600 baud will be correct, and the CPU clock will be slightly incorrect.

Do you want to store the new OSCCAL somewhere in the boot section ? That is possible. Then both the bootloader and the sketch could retrieve that value.

When you are sure that the ATmega chip will always be used at 25 degrees and 5V, you could calibrate a better value at 25 degrees and 5V, and store that in EEPROM or in the boot section.
But if someone uses that ATmega at freezing temperatures, then the baudrate will still be wrong.

I forgot to mention that a lower baudrate will not help in certain situations.

Suppose that the OSCCAL with the voltage and the temperature will make that the CPU clock speed has a 5% error. Then the baudrate has also a 5% error in speed. That is an additional error to the numbers at wormfood.net.

The 38400 and 9600 have both an error of 0.2% for the baudrate speed (because of the 8MHz and the hardware dividers).
With that 5% error, they will be both 5.2% ( or 4.8 ).
Therefor it does not matter if you choose 38400 or 9600, it makes absolutely no difference, they have the same error in the baudrate speed.
Even if you choose 300 baud, that 300 would also have a 5% error in the baudrate speed.

hansibull:
But what about the "correct" OSCCAL value? I don't want to hard code this into the bootloader either.

Do not do that. The value is specific to a batch.

Is there a reason you do not want to use a zero-error baud rate?

  1. The "correct" OSCCAL value will be different from one AVR to another, so you can't just hardcode a value into the bootloader. Presumably the "factory default value" is something that has been picked to be "close" for some batch of chips (per wafer, perhaps?)
  2. There are tools for "tuning" the OSCCAL value, but you normally have to run them manually.
  3. You could derive the most correct value automatically during the "burn bootloader" phase, and adjust a value in the bootloader "on the fly." An intriguing idea, but you'd need a special purpose bootloader burner.
  4. If you use the software bit-banged serial code, you might be able to have THAT be self-tuning regardless of the clock accuracy.
  5. It might be possible to have the bootloader self-tune OSCCAL based on the serial data rate. Maybe only the first time it communicates...

At least 1284 and 644 have a lot of room for extra code...

Don't forget that an "accurate" serial data divisor may be less important than one that matches the USB/Serial converter. (Atmel Xplained 328 Mini has a crystal that clocks both, but they can't communicate at 115200, for example.)

I'm not entirely convinced that all of the "failure to communicate" issues are because of bad clock rate. For example, the new "retry" code that showed up in avrdude somewhat recently appears to be useless for Arduino, because the auto-reset generation is OUTSIDE of the retry loop :frowning:

Is there a reason you do not want to use a zero-error baud rate?

You mean like 128000 ? Not all host software supports non-standard rates, without going to a lot of trouble.

I want to find a baud rate that works with all microcontrollers without having to calibrate the oscillator. 38400 baud doesn't work with some of my uC's, but work well with others. 9600 baud work on some, but not all of them..

Here's the Avrdude output when trying to upload a simple blink sketch on the ATmega8535 running at 8 MHz with 9600 baud (Not done any calibration):

avrdude: Version 6.0.1, compiled on Apr 14 2015 at 16:30:25
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "/Users/Hans/Documents/Arduino/hardware/MightyCore/avr/avrdude.conf"
         User configuration file is "/Users/Hans/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port                    : /dev/cu.wchusbserial620
         Using Programmer              : arduino
         Overriding Baud Rate          : 9600
         AVR Part                      : ATmega8535
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PA0
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom         4    20   128    0 no        512    0      0  9000  9000 0xff 0xff
           flash         33     6    64    0 yes      8192   64    128  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           lock           0     0     0    0 no          1    0      0  2000  2000 0x00 0x00
           calibration    0     0     0    0 no          4    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : Arduino
         Description     : Arduino

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x94

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x20

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0xfc

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x4c
         Hardware Version: -1553391364
         Firmware Version: 5304688.-1553391364

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x24

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x10

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x4c

avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x34
         Vtarget         : 322122178.4 V
         Varef           : 274157311.2 V
         Oscillator      : 0.000 Hz
         SCK period      : 368336.6 us


avrdude: stk500_getparm(): (a) protocol error, expect=0x14, resp=0x10
avrdude: stk500_initialize(): (b) protocol error, expect=0x10, resp=0x83
avrdude: initialization failed, rc=-1
         Double check connections and try again, or use -F to override
         this check.

avrdude: stk500_disable(): protocol error, expect=0x14, resp=0xfc

avrdude done.  Thank you.

Problem uploading to board.  See http://www.arduino.cc/en/Guide/Troubleshooting#upload for suggestions.

it is quite easy several misconceptions.

hansibull:
I want to find a baud rate that works with all microcontrollers without having to calibrate the oscillator.

quite easy to do. the key is "autobaud" sensing. ive used this with custom bootloaders on many different avrs with great success and its never let me down yet.

several misconceptions. first there IS a default oscal stored in each chip and fetched at reset. generally works for virtually all chips at the spec sheet nominal clock speed, temperature, and voltage with no need for calibration. some part have higher guaranteed accuracy that others. some as high as 1% but in all modern chips worst case adequate for serial io.

secondly any attempt at user calibration and storing a new value has little benefit over using the factory number if variation in these conditions is expected.

also note that lower baud rates have no advantage over higher because divisor error is insignificant compared to internal clock drift.

i have a routine that works with tiny or mega series bootloaders and doesnt care about cpu speed, temperature, or vcc. some versions even auto-select port and bit. in fact the same code works on such wildly different targets as tiny13, mega8, or mega2560 without recompiling. it has made life very easy over the years.

as i said the solution is auto-baud every boot, not calibrating and storing oscal for each new chip. easy because the upload program sends same initiate transfer character every time. the more similar consecutive bits in that character the more accurate but even just the start bit alone can work. no point in storing, just use the new value every session so changing conditions dont matter.

westfw:
5) It might be possible to have the bootloader self-tune OSCCAL based on the serial data rate.

Precisely how Tiny Tuner 1 works.

Maybe only the first time it communicates...

I believe one of the t85 bootloaders does exactly that. The value is stored in EEPROM then loaded by the bootloader.

westfw:
You mean like 128000 ?

125 k, 250 k, 500 k, 1 M for a 1 MHz, 8 MHz, or 16 MHz clock.

I use 1 M for my Unos.

john1993:
also note that lower baud rates have no advantage over higher because divisor error is insignificant compared to internal clock drift.

Wrong. Higher baud rates require Double Speed Operation which...

Note however that the Receiver will in this case only use half the number of samples (reduced from 16 to 8) for data sampling and clock recovery, and therefore a more accurate baud rate setting and system clock are required when this mode is used.

i dont think all of us are on the same wavelength. discussions of normal crystal baud rate errors and uart divisors are not relevant here. the internal clock may be off by 1ppm or 10000ppm so have little to do with normal atmel baud tables or discussions of divisor error percentage or uart mode.

what we are dealing with is continuous vcc/temperature clock variations and oscal granularity and stability. all compensated with auto-baud sensing. as koepel tried to explain, unlike wth crystal, there is less advantage to lowering baud rate. in fact using internal osc can allow much more reliable communication at 115k than a xtl which technically violates timing.

also off topic one of the reasons i prefer soft serial for most projects is the fact that it has twice the precision/granularity of hardware uart even in 2x. aside from the fact that you can have multiple ports and can use any pins.

So how can auto baud be implemented in the bootloader and in the boards.txt? The idea is to make it as simple as possible for the user. Are the possibility of a successful upload greater if I bump up the baud rate to 250000 compared to 9600?

hansibull:
Are the possibility of a successful upload greater if I bump up the baud rate to 250000 compared to 9600?

http://wormfood.net/avrbaudcalc.php?bitrate=9600%2C250k&clock=1%2C8%2C16&databits=8

0.0% vs 0.2% for the target running at 8 and 16 MHz. About the same.

Impossible to compare when the target is running at 1 MHz.

hansibull:
Are the possibility of a successful upload greater if I bump up the baud rate to 250000 compared to 9600?

no. theoretically there should be no difference. however in the real world many factors contribute to success and imo anything beyond 57k has little improvement in flash time but significantly more trouble with things like noise and skew due to cable capacitance and inductance. i think you were on the right track there back in your first posts.

this has nothing to do with bit timing error which as i said no longer enters into the equation once you leave the world of xtl timing. atmel baud tables mean nothing when using internal osc.

how can auto baud be implemented

Hmm. If we're talking about the HW BRG, then changing the value by 1 at the rates we're talking about changes the baud rate by about 6%:

 BAUD(UBRR) = Fcpu/(8*(UBRR+1)) = 1e6/(UBRR+1)
 Baud(15) = 62500 (+6.9%)
 Baud(16) = 58823 (nominal "57600")
 Baud(17) = 55555 ( -5.5%)

That SHOULD mean that when the osc is off by enough to prevent com from working (ie more than 5%), bumping it 1 in a particular direction ought to be enough to move into compatible range. And the characters received in each of the "just a bit too far off" cases in either direction should be predictable...

(Now, if you're using a SW bit-banged UART, you can probably get even closer, just by measuring the time between the first two rxdata transitions - you know what it should be...)

Although it would be better (for millis()/etc) to fix OSCCAL...