Problems storing large data in PROGMEM

Other Hardware Storage
1

Hi,

I have a Mega 2560 board and I am trying to create a really large lookup table in PROGMEM space. I have split my lookup table into 4 pieces of 5000 elements of 4 bytes each, so the total size would be less than 80 Kb while the board has 256 Kb of flash memory.
My problem is that the board doesn't seem to be able to handle this amount of data even though it has enough memory. The sketch simply doesn't run at all, it looks like the board is constantly rebooting. If I reduce the size of the data by 1 piece leaving about 60 Kb, it works fine.

I'm using Arduino IDE 0022 on Windows, and the sketch is simply printing the data in a loop.

I think it has something to do with 64 Kb limit for 16-bit pointers, but I don't understand why the code doesn't run at all. Am I correct assuming that progmem arrays are placed after the code?
Or is my board simply broken?

2

Interesting. I have no help to offer on this (yet). I am going to follow to see how this works out.
I recall reading elsewhere (last winter) that there was a problem using more than 128K on the 256K parts, I don't know if that is resolved now.

How big is your sketch?

3

It's 83526 bytes.

4

This kind of situation needs to be managed carefully because of the 16-pointer limitation. If your program has a function foo() that ends up at address 70000 and you call it from address 20000 then it won't reach address 70000, it will reach 70000-65536=4464, or basically a completely random place in your code and your sketch will crash.

You need to control more carefully where things are placed in memory. You want all your code in the first 64kB and the data in the area above 64kB. I would tackle this problem at a level lower than the Arduino IDE and middleware, using linker scripts and sections to directly place code and data where I want them. Perhaps someone has some links for getting started on this (I don't have any good resources at the moment).

--
The Aussie Shield: breakout all 28 pins to quick-connect terminals

5

Do you mean that Arduino IDE can't handle a sketch more than 64 Kb?
Where can I find the steps the IDE performs to compile and link the code?
I suppose that compiler should give a warning in this case if it can't generate a pointer. Do you think it might help if I get a newer gcc?

6

Is this just simply a matter of using the "far" variants of the progmem accessors, instead of "near"?

7

Yes, I am using *_far functions to access the memory. The problem is that even setup() function is never executed.

8

I'm attaching my sketch, can anyone with a Mega 2560 test it please?

progmem_test.zip (90.7 KB)

9

mysin.h.
Call me old-fashioned an picky, but a declaration for a 5000 element array shouldn't have 20000 lines of source, should it?

Edit: missed the breaks, scrolling too fast - sorry.

10

There are problems with the version of avr-gcc and avr-libc used in the Arduino IDE. I have not been able to make PROGMEM work for large structures.

C++ programs crash on reset if a jump table is above 0xffff

There is not good support for PROGMEM_FAR.

Here are some links that explain some problems

http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1274821710

http://www.avrfreaks.net/index.php?name=PNphpBB2&file=viewtopic&p=708225

11

I have just tried a newer gcc version, and the sketch runs very well with it. Isn't it a time to upgrade gcc in Arduino distribution?