Does optiboot handle loading of files > 64KB correctly?

Using Arduino Microcontrollers
1

Re the following thread, several of us been trying to get sketches with upload files > 64KB to
run on atmega1284 chips using maniacbug's optiboot, but with problems.

http://arduino.cc/forum/index.php/topic,139671.0.html

The files will upload and verify ok, but not run correctly. This was also found to be the case
for atmega2560 boards. So, we're wondering where the problem is - IDE toolchain, compiler,
avrdude, optiboot?

There may also be some problem with how our sketches are formulated - ie, in using large
PROGMEM arrays, cf reply #57,

http://arduino.cc/forum/index.php/topic,139671.45.html

So the "specific" question here is ... is it known if optiboot itself will correctly handle sketch
files > 64KB? Have people actually gotten these to load and run successfully?

2

I am not the author of Optiboot Optiloader, so I can't say for sure. However I did a rewrite to specifically handle the processors with more memory (such as the Mega) and this sketch works:

Clarification: Are you asking about Optiboot or Optiloader?

3

Optiboot

The issue is testing loading of very large test sketches into boards such as the 1284P boards of different flavors and into arduino mega1280/2560 boards which don't have a modern opitboot bootloaders. It would be great if your 'Atmega chip programmer' sketch had more bootloader options for the mega1280/2560 boards. Also creating large sketches has been an issue as apparently using PROGMEM has a avr-gcc bug that won't allow it to correctly create large arrays in flash memory, even though they compile without error.

Lefty

4

retrolefty:
The issue is testing loading of very large test sketches into boards such as the 1284P boards of different flavors and into arduino mega1280/2560 boards which don't have a modern opitboot bootloaders.

Lefty, what does that mean, "modern optiboot bootloaders" ?

My app (sketch) is much bigger than half the flash space in a 1284P and uses PROGMEM quite a bit. Optiboot loads and runs it perfectly.

5

Actually, as I dig deep into the dark recesses of my memory (a scary place) I do seem to recall that this was an issue with Optiboot and westfw and I had some communication about it, and it was fixed. I'll review my email, but I suggest you (1) make sure you have the latest optibiit and (2) review its source for comments about this issue. I'll do the same.

6

skyjumper:

retrolefty:
The issue is testing loading of very large test sketches into boards such as the 1284P boards of different flavors and into arduino mega1280/2560 boards which don't have a modern opitboot bootloaders.

Lefty, what does that mean, "modern optiboot bootloaders" ?

My app (sketch) is much bigger than half the flash space in a 1284P and uses PROGMEM quite a bit. Optiboot loads and runs it perfectly.

The existing mega2560 as well as the older mega1280 boards use a older bootloader the has known problems, no proper WDT support, !!! monitor triggering problem, and we are not sure they can handle uploading past a certain size >64KB. Optiboot fixes a few (all?) of those issues, or at least we are trying to test and validate that. As far as the PROGMEM problem, it's not using it that is a problem, but rather if used to create arrays past a certain size that is the apparent problem.

Lefty

7

skyjumper:
Actually, as I dig deep into the dark recesses of my memory (a scary place) I do seem to recall that this was an issue with Optiboot and westfw and I had some communication about it, and it was fixed. I'll review my email, but I suggest you (1) make sure you have the latest optibiit and (2) review its source for comments about this issue. I'll do the same.

Well locating a optiboot bootloader for my mega1280 board has been a problem for me. And I'm not sure that the arduinoISP programmer uses the proper AVRDUDE protocol (SK500v1 Vs SK500v2 ?) to load bootloaders into the high memory addresses that the mega1280/2560 boards use and I have access to no other hardware programmer other then my USBtiny that is known not to work with mega1280/2560 boards.

I think all the basic problems are known and have solutions or workarounds, I'm just not able to assemble every thing together to really make valid tests.

Any one have a link(s) to very large size arduino sketches ( say 100KB and 200KB that don't rely on using PROGMEM to attain their large size?

Lefty

8

OK, well we have two issues here. First is getting the bootloader onto the chip, which can be a problem with some programmers with > 64K words of program memory.

The second is, to have a bootloader that itself can handle the extra memory itself (effectively the same problem, moved to the bootloader).

I have seen so many incarnations of bootloaders, I am unsure which one work absolutely correctly. I believe the one which my bootloader-installer installs works correctly, but haven't tried a very large sketch (with that bootloader). I have tested the WDT issue though.

9

retrolefty:
Any one have a link(s) to very large size arduino sketches ( say 100KB and 200KB that don't rely on using PROGMEM to attain their large size?

I think mine is 80k or 90k, not sure. I have not looked in a while. It uses very little PROGMEM. I can send it to you.

10

retrolefty:

[quote author=Nick Gammon link=topic=143876.msg1080713#msg1080713 date=1358717944]
OK, well we have two issues here. First is getting the bootloader onto the chip, which can be a problem with some programmers with > 64K words of program memory.

The second is, to have a bootloader that itself can handle the extra memory itself (effectively the same problem, moved to the bootloader).

I have seen so many incarnations of bootloaders, I am unsure which one work absolutely correctly. I believe the one which my bootloader-installer installs works correctly, but haven't tried a very large sketch (with that bootloader). I have tested the WDT issue though.

After much searching yesterday and today I finally located a optiboot hex file compiled for a mega1280 board in someone's site. I just was able to burn it into two different mega1280 boards I have using arduinoISP sketch from IDE 1.0.3 as the programmer type and they both took the new optibootloader without error. I tested both with a WDT testing sketch that would 'brick' a mega board with it's original bootloader, but now function correctly, so yea.

So the present arduinoISP programmer sketch has no issues burning bootloaders into large memory AVR chips.

So if anyone else is having trouble finding a optiboot for a mega1280 chip I will post it here:

:020000000404F6
:020000021000EC
:10FC0000112484B714BE81FFF2D085E08093810077
:10FC100082E08093C00088E18093C10086E08093F9
:10FC2000C20080E18093C4008EE0CBD0279A86E0AA
:10FC300020E33CEF91E0309385002093840096BB55
:10FC4000B09BFECF1F9AA8958150A9F7CC24DD2444
:10FC500099249394A5E0BA2EF1E1AF2EA6D0813479
:10FC600061F4A3D0082FB3D0023811F0013811F499
:10FC700084E001C083E091D08DC0823411F484E12E
:10FC800003C0853419F485E0AAD084C08535A1F479
:10FC90008CD0082F10E089D0E82EFF24FE2CEE2413
:10FCA000E02AF12A8F2D881F8827881F8BBFEE0C32
:10FCB000FF1C8DD067016EC0863521F484E08FD0A3
:10FCC00080E0D9CF843609F042C06FD06ED0082FC3
:10FCD0006CD080E0C81680EED80620F483E0F601F0
:10FCE00087BFE895C0E0D2E060D089930C17E1F7B8
:10FCF000F0E0CF16F0EEDF0620F083E0F60187BFDC
:10FD0000E89565D007B600FCFDCFA601A0E0B2E003
:10FD10002C9130E011968C91119790E0982F8827C4
:10FD2000822B932B1296FA010C0197BEE8951124B1
:10FD30004E5F5F4FF3E0A030BF0751F7F601B7BE4B
:10FD4000E89507B600FCFDCFA7BEE89523C0843731
:10FD5000A1F42BD02AD0E82E28D039D0E6010E2DE0
:10FD6000FE0186911AD021960150D1F70894C11C4A
:10FD7000D11CEA94CE0CD11C0DC0853731F427D0AC
:10FD80008EE10BD087E909D075CF813511F488E079
:10FD900018D01DD080E101D061CF982F8091C00094
:10FDA00085FFFCCF9093C60008958091C00087FF27
:10FDB000FCCF8091C00084FD01C0A8958091C60051
:10FDC0000895E0E6F0E098E1908380830895EDDF08
:10FDD000803219F088E0F5DFFFCF84E1DECF1F939A
:10FDE000182FE3DF1150E9F7F2DF1F91089580E04B
:08FDF000E8DFEE27FF2709946C
:040000031000FC00ED
:00000001FF

I also had to create a new board entry for the boards.txt file in the arduino core:

##############################################################

megao.name=Arduino Mega1280 Optiboot
megao.upload.protocol=arduino
megao.upload.maximum_size=130048
megao.upload.speed=115200
megao.bootloader.low_fuses=0xff
megao.bootloader.high_fuses=0xda
megao.bootloader.extended_fuses=0xf5
megao.bootloader.path=optiboot
megao.bootloader.file=optiboot_atmega1280.hex
megao.bootloader.unlock_bits=0x3F
megao.bootloader.lock_bits=0x0F
megao.build.mcu=atmega1280
megao.build.f_cpu=16000000L
megao.build.core=arduino
megao.build.variant=mega

And finally a WDT testing sketch that someone on this forum wrote that works fine on a Uno type board but WILL BRICK any arduino mega1280/2560 board that hasn't been upgraded with the opitboot bootloader, so only run it if you know what your are doing and have the means to reburn the bootloader to recover.

// Test sketch to see if WDT interrupts are handled properly by the bootloader

/*
 Warning Warning Warning this is a semi-destructive test in that
 if your bootloader does not reset WDT interrupts upon starting 
 it will be forced into a tight loop of bootloader starts/WDT resets
 chip/bootloader starts again/lather rinse repeat. One can only
 recover to normal operation by reburning the bootloader with
 a ICSP programmer. If this sketch runs properly on a chip with a WDT aware bootloader,
 you will see continous serial output on the serial monitor.

Note that current arduino mega boards will fail this test and brick

*/

// Code from arduino forum poster (sorry name not remembered) 1/11/13

#include <avr/wdt.h>

void setup(){
  Serial.begin(57600);
  delay(100);
  Serial.println("Hello world");
  wdt_enable(WDTO_15MS);
}

void loop(){
  Serial.println("I am going to not get stuck..");
  for(int x=0; x<100; x++) {
    wdt_reset();
    x++;
    delay(10);
  }
  wdt_reset();
  Serial.println("I am going to get stuck now..");
  for(int x=0; 1; x++) {
    delay(10);
  }
}

Lefty
[/quote]

11

For my part, related to the first post, I've tried a Mega2560 R3 board that I bought just a
couple of months ago, and for the 1284, I think we've all been using maniacbug's latest
version of optiboot, from here,

Plus it's been tried with IDE 1.0, 1.03, 0022, and the enhanced IDE ERW 1.03. We are trying
to determine where the problem is, so started with asking about optiboot, per the first post
here.

12

It should be easy enough to check the bootloader functionality; just upload via the bootloader, and then run verify with an ISP programmer.

I was looking at memory regions a little while ago, thinking that I didn't understand how the compiled code was supposed to work if the amount of data in pgmspace got large. Perhaps it just... doesn't.
My understanding is that long jumps are supposed to work via "trampolines." A trampoline is essentially a jump table to the >64k target in the first 64k of memory, letting most things work with the 16bit pointers that the compiler likes to use. But if you have lots of pgmspace DATA, that's also located in the first 64k of memory (so that the pgm_read_byte() code can work, and I think it pushes out the jump table.

So sketches that have more than 64k of CODE might work a lot better than sketches with lots of data...

13

westfw:
It should be easy enough to check the bootloader functionality; just upload via the bootloader, and then run verify with an ISP programmer.

I was looking at memory regions a little while ago, thinking that I didn't understand how the compiled code was supposed to work if the amount of data in pgmspace got large. Perhaps it just... doesn't.
My understanding is that long jumps are supposed to work via "trampolines." A trampoline is essentially a jump table to the >64k target in the first 64k of memory, letting most things work with the 16bit pointers that the compiler likes to use. But if you have lots of pgmspace DATA, that's also located in the first 64k of memory (so that the pgm_read_byte() code can work, and I think it pushes out the jump table.

So sketches that have more than 64k of CODE might work a lot better than sketches with lots of data...

I think that is correct thinking. Now just to find some jumbo sketches the use code the nearly fills 128KB and 256KB chips.

Thanks;
Lefty

14

Ok, I have a bit of information. I created large sketches by simply cloning a couple of
digitalWrite() statements a gazillion times. I also cut back on the size of the PROGMEM
arrays to a total of 16 KBytes. Then uploaded to both 1284 and 2560 boards. In both
cases, the sketches appear to work correctly afterwards, as evidenced by Led13
blinking properly as directed by the code.

The mega1284 binary sketch size was 84,276 bytes, and I added more code, so the
mega2560 binary sketch size was 212,524 bytes.

So it looks like IDE v.103, plus optiboot for both 1284 [maniacbug] and Mega2560 R3
chips, can handle sketches > 64 KB alright. Apparently, what they didn't like before
were the PROGMEM arrays being too big.