New optiboot; beta testers welcome...

[westfw]

I've been working on the Arduino copy of optiboot, and have a new version that could use some testing.
It pretty much tries to fix all the optiboot-related issues open for Arduino:

• 380 optiboot has problems uploading sketches bigger than about 30 KB


• 517 Makefile in the /optiboot directory not functional with avrdude 5.10 / gnu make 3.8.1


• 556 update optiboot to the point of the latest optiboot project sources.


• 487 should be possible to compile optiboot using the Arduino-installed tools.


• 526 optiboot can start sketchs with inconsistent regster configuration side-effects


• 554 Optiboot source/binary should include a version number.


• 555 Optiboot high-value watchdog timeouts are defined incorrectly.


• Code space reduction (alas, more features show up to consume the saved space)


• Additional platforms.  Notably atmega8.


• 368 Optiboot does not support ArduinoasISP programmer


• Return FW version numbers in response to appropriate commands.


• The pre-built list of .hex and .lst files is reduced.

Although there are substantial numbers of changes, very little of the core logic of optiboot has changed.

Tested on m328, m168, m8, some with manual reset, and with ArdunioISP. Mostly on Mac, some on windows xp. Needs more linux testing.

Hex file (for m328) attached.  Source and stuff at https://github.com/WestfW/Arduino

[madworm]

Issue 517 seems fixed on my system, which is neither macos nor the unmentionable os.

Compiling using 'makeall' works as well. And the .hex file size is below 512 bytes. I'm using avr-gcc 4.4.3 + avr-libc 1.7.1 (from avrfreaks.net using Bingo's build script).

[nospam2000]

Hi westfw,

I've been working on the Arduino copy of optiboot, and have a new version that could use some testing.
Hex file (for m328) attached.  Source and stuff at https://github.com/WestfW/Arduino

Yesterday I started a new optiboot clone at google code which includes almost the same changes to the original optiboot you also mentioned  

My main focus was to integrate the BlueController board in the Arduino environment. The BlueController includes a Bluetooth module BTM-222 onboard (just like the Arduino-BT), but using my modified optiboot bootloader, the download of sketches works much more reliable as on the Arduino-BT!

Some additions were necessary for the BlueController which can be useful for any other boards which don't support auto-reset, even for the Arduino-BT. Currently I use the #define BLUECONTROLLER to distinguish between the BlueController board and other boards, but it makes sense to use multiple #defines for hardware specific code (like the LED pin number) and the support of the longer timeout for boards without auto-reset.

Is there a reason using GitHub instead of Google code? I wanted to merge back my changes to the original optiboot branch, therefore I used Google code. The most obvious advantage is that my project is directly linked together with the original project. Actually I wouldn't have started the new clone if I had seen yours before.

 Michael

[nospam2000]

Hi westfw,

• Code space reduction (alas, more features show up to consume the saved space)


• 380 optiboot has problems uploading sketches bigger than about 30 KB

the fix for issue 380 (see below) is consuming 32 additional bytes which I badly needed for my BlueController extensions. Therefore I could not use the overlapping boot_page_erase() and serial communication. For the Bluetooth communication this doesn't matter. Comparing the time with and without overlapping erase gave no difference. I haven't tested it with USB UART communication right now.

The code that could be saved is in bold:
+      // If we are in RWW section, immediately start page erase
+#if !defined(BLUECONTROLLER)
+      if (address < NRWWSTART)
+   1fcba:   80 e0          ldi   r24, 0x00   ; 0
+   1fcbc:   e8 16          cp   r14, r24
+   1fcbe:   80 ee          ldi   r24, 0xE0   ; 224
+   1fcc0:   f8 06          cpc   r15, r24
+   1fcc2:   20 f4          brcc   .+8         ; 0x1fccc <main+0xcc>
+        __boot_page_erase_short((uint16_t)(void*)address);
+   1fcc4:   83 e0          ldi   r24, 0x03   ; 3
+   1fcc6:   f7 01          movw   r30, r14
+   1fcc8:   87 bf          out   0x37, r24   ; 55
+   1fcca:   e8 95          spm
+   1fccc:   c2 e0          ldi   r28, 0x02   ; 2
+   1fcce:   d2 e0          ldi   r29, 0x02   ; 2
+#endif
[...]
+      // If we are in NRWW section, page erase has to be delayed until now.
+      // Todo: Take RAMPZ into account
+#if !defined(BLUECONTROLLER)
+      if (address >= NRWWSTART)
+   1fcd8:   f0 e0          ldi   r31, 0x00   ; 0
+   1fcda:   ef 16          cp   r14, r31
+   1fcdc:   f0 ee          ldi   r31, 0xE0   ; 224
+   1fcde:   ff 06          cpc   r15, r31
+   1fce0:   20 f0          brcs   .+8         ; 0x1fcea <main+0xea>
+#endif
+        __boot_page_erase_short((uint16_t)(void*)address);
+   1fce2:   83 e0          ldi   r24, 0x03   ; 3
+   1fce4:   f7 01          movw   r30, r14
+   1fce6:   87 bf          out   0x37, r24   ; 55
+   1fce8:   e8 95          spm
+      }
+#endif

Without loosing any performance or functionality, the code can be optimized to compare only the high-byte of the NRWWSTART, (because the low byte of NRWWSTART will always be zero) which saves 6 bytes:

Code: [Select]

     // If we are in RWW section, immediately start page erase
#if !defined(BLUECONTROLLER)
     if ((uint8_t)(address >> 8) < (uint8_t)(NRWWSTART >> 8))
   7eb0:       8d 2d           mov     r24, r13
   7eb2:       99 27           eor     r25, r25
   7eb4:       08 2f           mov     r16, r24
   7eb6:       80 37           cpi     r24, 0x70       ; 112
   7eb8:       20 f4           brcc    .+8             ; 0x7ec2 <main+0xc2>
       __boot_page_erase_short((uint16_t)(void*)address);
   7eba:       83 e0           ldi     r24, 0x03       ; 3
   7ebc:       f6 01           movw    r30, r12
   7ebe:       87 bf           out     0x37, r24       ; 55
   7ec0:       e8 95           spm
   7ec2:       c2 e0           ldi     r28, 0x02       ; 2
   7ec4:       d1 e0           ldi     r29, 0x01       ; 1
#endif      
[...]
     // If we are in NRWW section, page erase has to be delayed until now.
     // Todo: Take RAMPZ into account
#if !defined(BLUECONTROLLER)
     if (!((uint8_t)(address >> 8) < (uint8_t)(NRWWSTART >> 8)))
   7ece:       00 37           cpi     r16, 0x70       ; 112
   7ed0:       20 f0           brcs    .+8             ; 0x7eda <main+0xda>
#endif
       __boot_page_erase_short((uint16_t)(void*)address);
   7ed2:       83 e0           ldi     r24, 0x03       ; 3
   7ed4:       f6 01           movw    r30, r12
   7ed6:       87 bf           out     0x37, r24       ; 55
   7ed8:       e8 95           spm
     }
#endif


I don't understand why the compiler generates the mov+eor+mov instructions for the first compare, maybe you have an idea?

 Michael

[westfw]

Is there a reason using GitHub instead of Google code?

Yeah, the plan is to maintain the arduino-associated copy of optiboot separately as part of the arduino codebase (which is hosted on github.)  The tree I pointed to is forked off of the main arduino branch, and if all goes well it will be pulled back in and become the official code.

The NRWW section stuff is straight from the existing optiboot source tree (mercurial from google code); it (the optiboot google code repository) was 506 bytes to start with.  I got back some 28 bytes by moving address and length into registers, and used most of it up implementing the versioning, the ArduinoISP fix, and keeping the compiler happy (the new version is 502 bytes.)

I like your patch to only compare the high byte.

Does the extra code you were worried about go away if you simply define NRWWSTART to 0 ?   gcc is usually pretty good about optimizing away if clauses that are never true.  An attempt here says I get back 26 bytes, but I didn't look at the object code to see whether that was all that could be gotten...

[westfw]

Comparing the time with and without overlapping erase gave no difference.

Interesting.  I couldn't measure a difference on my FTDI-based Arduino's either, but on an Uno (with the AVR-based USB/Serial chip) there was about a 20% speedup using the overlap.

The Uno was over TWICE as fast as the FTDI-based Arduino, with everything else being the same.  (took the 328 out of the Uno and put it in a Duemilanove.) (with or without the overlap code.)

[nospam2000]

The Uno was over TWICE as fast as the FTDI-based Arduino, with everything else being the same

They are playing in a different league =(

The bluetooth communication of the BlueController and the Arduino BT is around 30 times slower than the direct serial communication, so an additional three seconds really don't matter.

When I have the time I will do some investigations about the problem. I think the reason is the much larger latency and not the throughput of the connection. Every ping-pong takes its time.

  Michael

[tim7]

Very happy to see this project given new life.  I'm just reading through the source changes before trying out the new version, and have a couple of questions:

• Am I right in thinking that the bootloader exits if it detects any corrupt serial communication, on the assumption that the baud-rate is wrong and therefore the data is not coming from a "normal" AVRdude upload?  Should we expect any communication data to be lost in this process, or will the application be able to re-read the corrupt character at the correct baud rate?
  


• I see the version numbers are put into the last two bytes of the bootloader block, and are now accessible both to the application code and via the serial port.  But how do I query the version numbers over the serial port?
  

Thanks very much for making these updates, and for making the code available.

[westfw]

# Am I right in thinking that the bootloader exits if it detects any corrupt serial communication, on the assumption that the baud-rate is wrong and therefore the data is not coming from a "normal" AVRdude upload?  Should we expect any communication data to be lost in this process, or will the application be able to re-read the corrupt character at the correct baud rate?

Actually, it must detect nothing but corrupt data for the entire bootloader timeout (~1s) to exit.  I'm not sure that that's a great algorithm, but it seems to be working OK.
The application is assumed to have some sort of retransmission mechanism of its own; once those characters have been read (or mis-read) by the bootloader, they're gone...

# I see the version numbers are put into the last two bytes of the bootloader block, and are now accessible both to the application code and via the serial port.  But how do I query the version numbers over the serial port?

It's part of the standard startup dialog that avrdude does.  You can get it typed out with a single verbosity level (highlighted in RED below):

/Applications/arduino/arduino-0022/Arduino-0022.app/Contents/Resources/Java/hardware/tools/avr/bin/avrdude -C/Applications/arduino/arduino-0022/Arduino-0022.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf -patmega328p -cstk500v1 -P /dev/tty.usbserial-A7007sjz -b115200 -v

avrdude: Version 5.4-arduino, compiled on Oct  9 2007 at 11:20:31
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/

         System wide configuration file is "/Applications/arduino/arduino-0022/Arduino-0022.app/Contents/Resources/Java/hardware/tools/avr/etc/avrdude.conf"
         User configuration file is "/Users/billw/.avrduderc"
         User configuration file does not exist or is not a regular file, skipping

         Using Port            : /dev/tty.usbserial-A7007sjz
         Using Programmer      : stk500v1
         Overriding Baud Rate  : 115200
         AVR Part              : ATMEGA328P
         Chip Erase delay      : 9000 us
         PAGEL                 : PD7
         BS2                   : PC2
         RESET disposition     : dedicated
         RETRY pulse           : SCK
         serial program mode   : yes
         parallel program mode : yes
         Timeout               : 200
         StabDelay             : 100
         CmdexeDelay           : 25
         SyncLoops             : 32
         ByteDelay             : 0
         PollIndex             : 3
         PollValue             : 0x53
         Memory Detail         :

                                  Block Poll               Page                       Polled
           Memory Type Mode Delay Size  Indx Paged  Size   Size #Pages MinW  MaxW   ReadBack
           ----------- ---- ----- ----- ---- ------ ------ ---- ------ ----- ----- ---------
           eeprom        65     5     4    0 no       1024    4      0  3600  3600 0xff 0xff
           flash         65     6   128    0 yes     32768  128    256  4500  4500 0xff 0xff
           lfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           hfuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           efuse          0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           lock           0     0     0    0 no          1    0      0  4500  4500 0x00 0x00
           calibration    0     0     0    0 no          1    0      0     0     0 0x00 0x00
           signature      0     0     0    0 no          3    0      0     0     0 0x00 0x00

         Programmer Type : STK500
         Description     : Atmel STK500 Version 1.x firmware
         Hardware Version: 3
         Firmware Version: 4.4
         Vtarget         : 0.3 V
         Varef           : 0.3 V
         Oscillator      : 28.800 kHz
         SCK period      : 3.3 us

avrdude: AVR device initialized and ready to accept instructions

Reading | ################################################## | 100% 0.02s

avrdude: Device signature = 0x1e950f

[nospam2000]

Actually, it must detect nothing but corrupt data for the entire bootloader timeout (~1s) to exit.  I'm not sure that that's a great algorithm, but it seems to be working OK.

No, one character which doesn't fit in the protocol is enough:

Code: [Select]


### default case of main loop:
else {
      // This covers the response to commands like STK_ENTER_PROGMODE
      verifySpace();
    }

### will start app if the next character is != CRC_EOP
void verifySpace() {
  if (getch() != CRC_EOP) {
    watchdogConfig(WATCHDOG_16MS);    // shorten WD timeout
    while (1)                         // and busy-loop so that WD causes
      ;                               //  a reset and app start.


This is one of my changes. As long as the initial sync sequence has not been detected, all errors are ignored:

Code: [Select]

#if defined(BLUECONTROLLER) 
    if(ch == STK_GET_SYNC) {
      // this is the initial sequence, sent by avrdude
      verifySpace();
      blueCAvrdudeSynced = 1; // ignore all errors as long as this is 0
    }
    else 
#endif

void verifySpace() {
  if (getch() == CRC_EOP)
  {
    putch(STK_INSYNC);
  }
  else
  {
#if defined(BLUECONTROLLER)
    // ignore error when not synced, otherwise some initial garbage will exit the bootloader
    if(blueCAvrdudeSynced)
      appStart();
#else
    appStart();
#endif
  }
}


[leo72]

Needs more linux testing.

Hello. These are my 2bits for Linux.

Your precompiled .hex works perfectly with my UNOr1 under Xubuntu 11.04: I can use ArduinoISP without using additional components (caps, res, etc..).

I then tried to compile the bootloader from sources and it seems to me that everything was OK (output is in the attachment)
Thank you for your work and ask me if you need some more feedbacks

[westfw]

No, one character which doesn't fit in the protocol is enough:

Ah.  Depending on where it is.  The bootloader will accept (nearly) any "command" characters without exiting, but as you say is very particular about getting the EOP in the correct place...

The reason that ArdunioISP wasn't working was that garbage characters (received at the wrong bitrate) were causing the bootloader to keep trying, instead of going on and starting the sketch (ArduinoISP itself.)  I think your patch would make that worse, assuming the watchdog is still always reset in getch().  Normally, getting to the sketch promptly when the character stream doesn't look like bootloader commands is a good thing.   Any ideas on resolving this incompatibility?

[nospam2000]

I think your patch would make that worse, assuming the watchdog is still always reset in getch().  Normally, getting to the sketch promptly when the character stream doesn't look like bootloader commands is a good thing.   Any ideas on resolving this incompatibility?

The watchdog is reset in getch() even when waiting for but not getting a new character, but there is a configurable time limit based on timer overflow. One timer overflow takes about 4.2s@16MHz, so the app would start after this timeout.

It's too late to think about it today, I'll be back in a week.

  Michael

[tim7]

Thanks for the explanations.  For me the new bootloader seems to be working fine.  I have also tested it with ArduinoISP, without disabling auto-reset.

• Compiled under Windows-7 64-bit with the IDE v0022 tools


• Uploaded optiboot_atmega328.hex to an Arduino Uno, via Optifix running on another Uno


• Uploaded optiboot_atmega328.hex to an Arduino Uno, via IDE and ArduinoISP running on another Uno


• Uploaded optiboot_atmega328.hex to a 5v 16MHz Ardunio Pro


• Uploaded optiboot_atmega328_pro_8MHz.hex to a 3.3v 8MHz Ardunio Pro


• All boards now report Firmware Version: 4.4 to AVRdude, and correctly accept sketch uploads from the IDE

The 8MHz and 16MHz Pro boards were tested using an FTDI USB-to-serial interface.  As with previous version of Optiboot, for the 8MHz boards I needed to reduce the baud rate to 57600 in the Makefile and in boards.txt.

[JChristensen]

Using this bootloader fixes the problem (well, for me, anyway) of not being able to upload sketches on the occasional ATmega328P chip that I and one or two others have experienced.  See this post for a description of the problem.