possible error in optiboot.c ... ???

I was looking at the code in optiboot, and found the following lines, which seem to me to be in error, although in practice they work correctly. I don't know if this had been noted and possibly revised since IDE v.1.05.

    register uint8_t length; 
.......

    /* Write memory, length is big endian and is in bytes */
    else if(ch == STK_PROG_PAGE) {
      // PROGRAM PAGE - we support flash programming only, not EEPROM
      uint8_t *bufPtr;
      uint16_t addrPtr;

      getch();			/* getlen() */
      length = getch();
      getch();

      // If we are in RWW section, immediately start page erase
      if (address < NRWWSTART) __boot_page_erase_short((uint16_t)(void*)address);

      // While that is going on, read in page contents
      bufPtr = buff;
      do *bufPtr++ = getch();
      while (--length);

Basically, what happens is the data that comes from avrdude right after the STK_PROG_PAGE char [ie, 0x64, 'd'] are first the high-byte and then the low-byte of the length of the page-write block. As is obvious here, the high-byte is ignored.

Also, looking back at the original ATmegaBOOT_168.c bootloader, the same lines read as follows, showing that both length bytes used to be intercepted.

	/* Write memory, length is big endian and is in bytes  */
	else if(ch=='d') {
		length.byte[1] = getch();
		length.byte[0] = getch();
		flags.eeprom = 0;
		if (getch() == 'E') flags.eeprom = 1;
		for (w=0;w<length.word;w++) {
			buff[w] = getch();	                        // Store data in buffer, can't keep up with serial data stream whilst programming pages
		}

In short, as written optiboot will work ok for page lengths <= 256, since a zero actually causes the do...while loop to execute 256 times, given the way it is written. In reality, for chips like the 1284 and 2560, avrdude sends 0x01,0x00 as the page length value, and optiboot ignores the 0x01, and uses just the 0x00, which works in the do...while loop the same as a length=256 would have worked.

I guess it's ok, as long as the page-write lengths never exceed 256 bytes, but .... ummmm.

Well spotted! Generally, more eyeballs = less bugs.

http://www.atmel.com/Images/doc2525.pdf

Page 11: The data block size should not be larger than 256 bytes.

LOL, that kind of requires a double-take. Table 17 shows use of high-byte,low-byte to encode block size, so although first gush has you thinking byte, a 256 codes as a word = 0x0100, sent as 0x01,0x00, which is what is being sent by avrdude for 1284 and 2560 chips.

EDIT: BTW, as an aside, do you know if there's a way to use the Arduino IDE to compile the boot loader .C source files, or do I have to do a separate install of AVR Studio?

oric_dan:
EDIT: BTW, as an aside, do you know if there's a way to use the Arduino IDE to compile the boot loader .C source files, or do I have to do a separate install of AVR Studio?

The IDE should have no problem compiling the source file. However, the IDE will not compile the source file with the correct compile options and it most certainly will not use the correct link options so the image will be unusable.

A makefile is included with Optiboot that works correctly with Arduino 1.0.x. Under Windows just navigate to the correct directory and run omake. There are a handful of parameters for tuning the image. This is an example from my computer where I built Optiboot with: no LED, 1000000 baud rate, m1284 processor...

C:\Arduino\arduino-1.0.5\hardware\maniacbug-mighty-1284p-68ed99c\bootloaders\optiboot >omake LED_START_FLASHES=0 BAUD_RATE=1000000 atmega1284

C:\Arduino\arduino-1.0.5\hardware\maniacbug-mighty-1284p-68ed99c\bootloaders\optiboot >..\..\..\tools\avr\utils\bin\make OS=windows ENV=arduino LED_START_FLASHES=0 BAUD_RATE=1000000 atmega1284
BAUD RATE CHECK: Desired: 1000000, Real: 1000000, UBRRL = 1, Error=0.0%
../../../tools/avr/bin/avr-gcc -g -Wall -Os -fno-inline-small-functions -fno-split-wide-types -mshort-calls -mmcu=atmega1284p -DF_CPU=16000000L  -DBAUD_RATE=1000000 -DLED_START_FLASHES=0      -DBIGBOO
T     -c -o optiboot.o optiboot.c
optiboot.c:307: warning: 'flash_led' declared 'static' but never defined
../../../tools/avr/bin/avr-gcc -g -Wall -Os -fno-inline-small-functions -fno-split-wide-types -mshort-calls -mmcu=atmega1284p -DF_CPU=16000000L  -DBAUD_RATE=1000000 -DLED_START_FLASHES=0      -DBIGBOO
T   -Wl,--section-start=.text=0x1fc00 -Wl,--section-start=.version=0x1fffe -Wl,--relax -nostartfiles -nostdlib -o optiboot_atmega1284p.elf optiboot.o
../../../tools/avr/bin/avr-size optiboot_atmega1284p.elf
   text    data     bss     dec     hex filename
    446       0       0     446     1be optiboot_atmega1284p.elf
../../../tools/avr/bin/avr-objcopy -j .text -j .data -j .version --set-section-flags .version=alloc,load -O ihex optiboot_atmega1284p.elf optiboot_atmega1284p.hex
..\..\..\tools\avr\bin\avr-objdump -h -S optiboot_atmega1284p.elf > optiboot_atmega1284p.lst
rm optiboot_atmega1284p.elf optiboot.o

C:\Arduino\arduino-1.0.5\hardware\maniacbug-mighty-1284p-68ed99c\bootloaders\optiboot >

Cool and thanks, I'll try it tomorrow during wake time. Dealing with those paths is a real dog. I see you put the mighty maniac library in the IDE folder, instead of in the sketch folder.

As another aside, I'm cooking up a way to spoof avrdude so I can use it reliably for RF uploading. Halfway there.
[time to crash now].

oric_dan:
I see you put the mighty maniac library in the IDE folder, instead of in the sketch folder.

I tend to put "production" cores with the IDE and "development" cores under the Sketchbook.

As another aside, I'm cooking up a way to spoof avrdude so I can use it reliably for RF uploading.

Have you searched for an existing upload protocol designed for radios? For example, I worked for a company that designed an AVR upload protocol specifically for CANBUS. It was a much better solution than trying to smash one of the Atmel programming protocols onto a CANBUS connection.

While you are pursuing the subject, bear in mind that the IDE has support for "custom" upload utilities. You are not constrained to use avrdude.

for chips like the 1284 and 2560, avrdude sends 0x01,0x00 as the page length value

Oh! That's annoying.
https://code.google.com/p/optiboot/issues/detail?id=104

Oh! That's annoying.

No doubt! (It's oric_dan's fault.)

Darn, secret is out. When it comes to the "powers-that-be", I always try to be as annoying as possible. Helps keep people awake.

[quote author=Coding Badly link=topic=241103.msg1729206#msg1729206 date=1400486324]

oric_dan:
I see you put the mighty maniac library in the IDE folder, instead of in the sketch folder.

I tend to put "production" cores with the IDE and "development" cores under the Sketchbook.

As another aside, I'm cooking up a way to spoof avrdude so I can use it reliably for RF uploading.

Have you searched for an existing upload protocol designed for radios? For example, I worked for a company that designed an AVR upload protocol specifically for CANBUS. It was a much better solution than trying to smash one of the Atmel programming protocols onto a CANBUS connection.

While you are pursuing the subject, bear in mind that the IDE has support for "custom" upload utilities. You are not constrained to use avrdude.[/quote]

Thanks again. I got the bootloader recompiles working now. In regards RF-uploading, over the past year and a half, I've tried several of the schemes found on the web [starting with adafruit] with no real success, and also tried getting people around here who are knowledgeable with avrdude and the bootloaders interested, but also with no success either. So, I decided to do it myself.

There are some systems [for sale] that use special hardware and custom software, but I wanted to use standard XBee radios and try to stay within the Arduino environment, and also not have to write software for Windows.

I think I have it about figured out now. The avrdude source is about impossible for me to decipher, but optiboot is pretty simple, and I grabbed a .HEX file for a sketch, and decoded what happens during uploading using the IDE compile window display. (that's where the topic of this thread came from). The transfers are really pretty straightforward, and how they get hosed by the XBee radio transmissions is fairly evident, I think. So, I think it can be spoofed - we'll see. I'll keep the other things you mentioned in mind, however. Thanks.

In regards RF-uploading, over the past year and a half, I've tried ... getting people around here who are knowledgeable with avrdude and the bootloaders interested, but also with no success either.

Sorry. I keep meaning to do more with wireless and networking, but it seems to keep getting put off...

The transfers are really pretty straightforward, and how they get hosed by the XBee radio transmissions is fairly evident

Well? Don't keep us in suspense! Publish your suspicions!

Basically, figured it out yesterday about time I started this thread, by looking at the upload data stream in the IDE compile window. It's avrdude's fault. I was planning to get my spoof to work, and then explain it, lol.

Ref the do...while loop shown above from optoboot. Avrdude sends over either 0x0080 or 0x0100 for the page write-length value (for the std mega chips), immediately followed by a completely "non-throttled" page of that many bytes in a continuous stream. The link has to keep up, or the data is lost, and there is no way built into avrdude to recover.

So, it looks like what happens at least with XBee is you're overrunning the data cache inside the radio. I guessed this a year or so ago, but didn't understand enough about the bootloader and transfer protocol to know how to deal with it. Shouldn't be all that big a deal to fix. I have my spoofer about half written now, so I should know pretty soon.

westfw:

for chips like the 1284 and 2560, avrdude sends 0x01,0x00 as the page length value

Oh! That's annoying.
Google Code Archive - Long-term storage for Google Code Project Hosting.

You will also notice that the PAGE_READ operation has the same iffy loop structure as PAGE_WRITE.

As an ADDENDUM to previous comments, I have to say I am totally underwhelmed by the lack of robustness of the avrdude upload protocol. Totally..... If the internet used something that poorly conceived, we'd never get an email or webpage across the net.

I am beginning to understand the protocol now, although trying to understand how avrdude executes it would take me weeks, I think, trying to decode 30+ .h files, plus another 30+ .c files. Having separately-compilable source files in C is nice, but most programs end of looking like a big plate of interlinked sphaghetti strands. Sheesh.

I have to give the avrfreaks guys [I think] and Brian Dean immense credit for building avrdude, as it must have taken weeks if not months of time to code. Unfortunately, at the highest-most conceptual level, the protocol was poorly conceived.

The basic problem is it just blasts off large blocks of data, up to 256 bytes, with no checksums or ability to throttle them, and no ability to check if each block is individually received properly. The only checks come after the ENTIRE program has been uploaded, and by then it's far too late to do anything about transmission errors. All in all, not the most robust transmission scheme.

Besides that, what I have found all along is that the weakest link is the sync attempt at the very beginning of the upload, where avrdude sends the [30], [20] commands. This is exactly where most of my upload failures occur, via hardwire, and before the page-write block-transfers even begin.

avrdude: Version 5.11, compiled on Sep  2 2011 at 19:38:36
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "C:\winapps\ArduinoIDE\arduino-1.0.5-windows\hardware/tools/avr/etc/avrdude.conf"

         Using Port                    : \\.\COM4
         Using Programmer              : arduino
         Overriding Baud Rate          : 115200
avrdude: Send: 0 [30]   [20] 
avrdude: Send: 0 [30]   [20] 
avrdude: Send: 0 [30]   [20] 
avrdude: Recv: 
avrdude: stk500_getsync(): not in sync: resp=0x00

avrdude done.  Thank you.

And which should look as follows, with [14], [10] acks coming back from the bootloader.

avrdude: Version 5.11, compiled on Sep  2 2011 at 19:38:36
         Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
         Copyright (c) 2007-2009 Joerg Wunsch

         System wide configuration file is "C:\winapps\ArduinoIDE\arduino-1.0.5-windows\hardware/tools/avr/etc/avrdude.conf"

         Using Port                    : \\.\COM4
         Using Programmer              : arduino
         Overriding Baud Rate          : 115200
avrdude: Send: 0 [30]   [20] 
avrdude: Send: 0 [30]   [20] 
avrdude: Send: 0 [30]   [20] 
avrdude: Recv: . [14] 
avrdude: Recv: . [10] 
         AVR Part                      : ATMEGA328P
         Chip Erase delay              : 9000 us
         PAGEL                         : PD7
         BS2                           : PC2
         RESET disposition             : dedicated
         RETRY pulse                   : SCK
         serial program mode           : yes
         parallel program mode         : yes
         Timeout                       : 200
         StabDelay                     : 100
         CmdexeDelay                   : 25
         SyncLoops                     : 32
         ByteDelay                     : 0
         PollIndex                     : 3
         PollValue                     : 0x53
         Memory Detail                 :
..............

For some reason, when it fails initially, avrdude can't get the attention of the bootloader, after what appears to be a feeble attempt, and gives up straightaway. If I can get that working better, maybe the rest will be doable.

Anybody got an idea? My guess is avrdude isn't waiting long enough for the acks to come back from the bootloader, possibly because of the time it takes the mega chip to boot up after the reset line is released. Just a guess.

To wit, by controlling the reset pin manually, if I release it at precisely the right time, not too soon and not too late, then the upload will take place, otherwise no dice. There is a very small window there for avrdude to catch hold of things to its own liking. Very finicky operation.

Bear in mind the protocols are designed for high speed manufacturing where communications problems would have been reduced / eliminated out of necessity.

[quote author=Coding Badly link=topic=241103.msg1733717#msg1733717 date=1400721666]

Bear in mind the protocols are designed for high speed manufacturing where communications problems would have been reduced / eliminated out of necessity.[/quote]

I should imagine that's the usual rationale, however bootloaders use serial comms links, and there have been fairly robust file transfer protocols for decades, eg, kermit, x-modem, y-modem, and a dozen others. Small packet transfers with checksums and handshakes, etc.

In any case, got any ideas on the reset-sync problem? Any -options in avrdude to change the sync-up business?

I am totally underwhelmed by the lack of robustness of the avrdude upload protocol.

It has nothing to do with avrdude, which supports MANY protocols (which is why its code is so hard to understand). This is the STK500v1 protocol developed by Atmel. (for the STK500 eval/prototype/programmer board, which WAS one of their really early products.)
It does suck; you'd think that such a simple protocol would at least be efficient and unambiguous to implement, but that's not the case either. And on top of that, it's a "programmer" communications protocol rather than a bootstrap protocol, which means it's full of command inappropriate to interacting with a bootloader.

The communications robustness was fixed (or addressed, anyway) in version2 of the protocol (stk500v2.) Stk500v2 is used on the MEGA and MEGA2560 boards; the smallest v2 implementation I know of is 4k bytes. Which is a bit much for the 328 :frowning:
I doubt whether it's possible to do a 512byte bootloader with an error-checking protocol. (although WRT wireless, some relatively small changes might help a lot.)

For all the failings of the protocol, the arduino bootload process seldom has communications errors. After all, it has a separate high-speed error-checked link to within a couple of cm of the microcontroller pins.

Good information. I'll check out the mega bootloader code. Is 512 bytes the maximum size for 328 chips? What about 1284?

I doubt whether it's possible to do a 512byte bootloader with an error-checking protocol.

I've been thinking about this. It really doesn't take much more code to receive a series of smaller blocks and send back an Ack_byte after each block is received. Little more than a loop within the existing page-write loop. Maybe a couple of lines to compute checksum.

Of course, you still have to get the PC end to comply.

I was just wondering if anything ever came out of the "Bug Notice" given in post #1 here, or whether it just fell into the cracks of Arduino-Land.

In the optiboot source, it was fixed in 6.0: Google Code Archive - Long-term storage for Google Code Project Hosting.
And then optimized in 6.2: Google Code Archive - Long-term storage for Google Code Project Hosting.

The Arduino.inc people haven't been very enthusiastic about pulling a new version of optiboot into the IDE distribution; I can see their point: most of the changes have been for non-arduino platforms.

Thanks.