Show Posts
Pages: [1]
1  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 05, 2013, 10:29:12 pm
OK, the following works:

Using the multi-progmem-segment method

1) Store 64k of strings in PROGMEM_SEG2  (doesn't have to be exact or padded) denoted with a memory start location of 0x20000
2) Store ~64K of strings in PROGMEM_SEG3  denoted with memory start location of 0x30000
3) Store 20k of pointers to those strings in the lowest progmem space with PROGMEM_FAR which goes...I dunno where.

Plz note that you cannot deposit into memory a chunk of data larger than 64k, at least not with this version of avr-gcc/ld in the arduino 1.0.3.  I believe this will be different once we get to avr-gcc V4.7...someday...maybe.


The strings can be accessed thus:

Code:

#include <morepgmspace.h>

if(accessing a char* in a string at a location < size of the 64k of stuff in seg 2){

      unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[0]) +wordNumber*2); //wordNumber is the index into the array of strings called dictionary0
      strcpy_PX(newWord,index,PROGMEM_SEG2_BASE);
  
    }
    else {
      unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[0])+wordNumber*2); //note that the array dictionary0 also has data in SEG3!
      strcpy_PX(newWord,index,PROGMEM_SEG3_BASE);

    }

char* strcpy_PX(char* des, uint_farptr_t src,unsigned long base)
{
  unsigned long p = base + src;

  char* s = des;
 
  do {
    *s = pgm_read_byte_far(p++);
  }
  while(*s++);
  return des;
}



Note that you cannot access the data in those SEG2/SEG3 locations through specification of an indexed addressing scheme like

Code:
unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[i]);

I have been utterly unsuccessful in storing and reading out of the SEG 1 location which is denoted with a memory start location of 0x10000.  However, I can put 20k of char*s at wherever the attr PROGMEM_FAR puts it.  I have to go back to look at the linker script (avr6.x) to see where that's being mapped to.  I know it's going into .data (or maybe just past it...)

At least I understand what's happening now.   If I try to put anything into PROGMEM_SEG1 I not only get errors trying to read it back, but the bootloader gives me a verification error - which is probably the root cause of the whole problem.  I don't have it in me to look into the bootloader at this point...  On to another problem!

Cheers to all
Joe
2  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 05, 2013, 01:52:57 am
Hi and thanks all,
I did get my code to work tonite. I divided the data and pointers into 3 segments each of which is less than 64k.  There are 2 PROGMEM segments containing a bunch of explicitly initialized strings of the form:
Code:
#include <morepgmspace.h> // and associated mods to ldscript to accommodate the PROGMEM attrs
const char abcde[9] PROGMEM_SEG2 = "aardvark";

And also

Code:
const char fghij[4] PROGMEM_SEG1 = "ant";

And another segment full of pointers to those strings like this

Code:
const char* dictionary[10000] PROGMEM_SEG3= {abcde,fghij,..};

I can pull the chars out of the various segments like this

Code:
char* myWordInSeg2= GET_FAR_ADDRESS(dictionary[0]) + SEG2_OFFSET + index*sizeof(char*);
       strcpy_Pseg2(localCharArray,myWordInSeg2); // I wrote the copy to handle addressing from SEG a

It works but I'm durned if I can figure out why...somewhere some 18/24 bit addressing is taking place via ELPM... I may just be lucky because my strings are all loaded sequentially. The char ptrs in SEG3 seem to me to be useless. I think I'm just getting into seg2 and yanking out strings wantonly without guidance from the ptrs in the dictionary  array. But somehow this is working. The fact I don't understand it makes me nervous it is not a rigorous solution that only works temporarily.

Anyway.. Onward to Arduweenie land,
Joe
3  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 04, 2013, 06:04:49 pm
Hi Nick
Yes, I have seen in the code <pgmspace.h> and also <morepgmspace.h> where for machines that support ELPM, the macros pgm_read_xxx_far are mapped into ELPM calls instead of LPM.  
However, I've yet to determine exactly how to use those.   It does seem to work just fine when I have a single 64k block of flash data tagged with the attribute PROGMEM.   I think this is true because the following works:

Code:
#include <pgmspace.h>
//...(other includes)
char myString[10];
const char abc[] PROGMEM = "abc";
const char def[] PROGMEM = "def";
const char* abcdef[] PROGMEM = {abc,def};
// ... stuff
for(int i=0;i<2;i++){
strcpy_P(myString,pgm_read_word(&abcdef[i]));
Serial.println(myString);
}
Note the use of strcpy_P is defined in <pgmspace.h>.   Note also that I can address the array with a var, which does not seem to be true if you try to create a var in memory space mapped through other means - like the terms PROGMEM_FAR and PROGMEM_SEGx as specified in <morepgmspace.h>   In those cases,  the compiler seems to want a static situation where you do something like;
Code:
 
long_address = GET_FAR_ADDRESS(abcdef) + BASE_ADDRESS_OF_MEMORY_SECTION + (which element)*sizeof(int);
strcpy_whatever(myString, pgm_read_word(long_address);

I suppose 24-bit addressing must exist (at least, 18-bit addressing must) exist even to accept an address in that upper 256k...

If I use multiple 64k flash sections or I just try to rely on 24-bit addressing by specifying
Code:
#include <morepgmspace.h>
const variable myVar[70k's worth]... a whole lot of stuff more than 64k
then this does not work...
Code:
int abc = pgm_read_word(&abcdef[i]);

So I am unsure of the occasions when 24-addressing is supported by the current compiler/linker, and when it's not...  As I said, it seems that with versions avr-gcc 4.6.2 and 4.7.2 it does not balk at the code which calls more than 64k.  But I haven't yet figured out if I can actually make it work with my app.  I still have to go back and de-Arduino-ize my Arduino code into pure AVR code to try that in another environment, like AVR-Eclipse or AVR-XCode, which support those compiler versions...

Cheers
Joe
4  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 04, 2013, 12:18:57 pm
...from above

But the important thing to note is that PROGMEM designates that the linker put your data into a space that is 64k in size.  If you try to specify data bigger than 64k it might actually compile, but your results are going to be irratic - at least mine were using 1.0.3.  Another important thing to note is that the PROGMEM attribute puts the data in a certain place, and only in that place.  If you have more than 64k of data, you can put some where PROGMEM says and that spot is 64k big, but then where do you put the rest, and how do you get to it?

Now, just for the sake of discussion - there are lots of flash memory sections, used for different things.  You have the .text section, and the .data section, and lots of .fini1, .fini2,.finix...etc.  Code and data is marked by the __attribute__((section(".blah"))) tag, and then there is a corresponding note in the linker script that says what to do with things that are in section ".blah".  

Some have had success putting data in the section marked with the attribute ".fini7".   The AVR documentation says this is a user-definable section, and you can certainly use it to put a 64k chunk. If you have more than 64k you need more than one section.

Well, after much weeping and gnashing of teeth I got a lot of help from this:

http://www.avrfreaks.net/index.php?module=PNphpBB2&file=viewtopic&t=93874&highlight=

which explains how to set up several PROGMEM memory segments/sections. It pushes the static data into flash above the code itself, as is recommended by many.  It also provides some code to address those sections.  So, using that method which not only required using an include called  "morepgmspace.h" - and plz note this is a modified one from Carlos Lamas's original one, as well as making mods to the linker script found as  "avr6.x" - which is the script used for chips that have 256k of Flash.  Other chips will have different linker scripts.

With that combination of things,  I could specify constant global vars with the tags:  PROGMEM_SEG1, PROGMEM_SEG2, PROGMEM_SEG3.  I can them address them separately through a variety of means.  The linker script is modified to place the data in locations which begin at 0x10000, 0x20000, and 0x30000 respectively.  

Given that combo of linker script changes and includes I could do the following in code given the way I specified the data above:

Code:
 
char wordIGot[50];
uint_farptr_t theUltimateAddress  = GET_FAR_ADDRESS(myDictionary) + PROGMEM_SEGyadda_BASE_ADDRESS + (indexOfWord*sizeof(char*));

strcpy_Pyadda(wordIGot,pgm_read_byte_far(theUltimateAddress);


Note that I have not yet figured out which strcpy function to use with these addresses, so I wrote my own.  

apparently, this also works in some universe but I have not yet got it to work:

Code:
#include <morepgmspace.h>
char myWord[50];
int theIndexOfTheWordIWant;
strcpy_PF(myWord, pgm_get_word_far(&myDictionary[theIndexOfTheWordIWant]));

What I have been unable to accomplish at this point, is that I cannot retrieve the char* pointers from the array of 10000 pointers I specified as "dictionary[]" above.
Just for sake of example, I have stored 64k of strings in the memory noted with the attribute PROGMEM_SEG2.  I have stored the 10000 pointers to that data in PROGMEM_SEG1.

I am presuming these char* pointers are all 16-bit, so that they are addresses to the char strings all WITHIN  the same memory segment, and that I will have to do something of the order of:

 
Code:
#include <morepgmspace.h> // note this is the MODIFIED one available on avrfreaks.com, not the original from Carlos Lamas's website
uint16_t theLocalAddress = pgm_read_word_far(&dictionaryWhichIsInSeg1[which word do I want]);

//but as I mentioned the above line doesn't seem to work always, as you can't retrieve the dictionary address through variable indexing.
//In that case, you have to do this..

 uint16_t theLocalAddress = pgm_read_word_far(GET_FAR_ADDRESS(dictionary) + PROGMEM_SEG1_BASE + whichWordDoIWant*sizeof(char*));

//but I'm not sure how well that works either


then

Code:
char* theRealAddress = pgm_read_word_far(theLocalAddress + PROGMEM_SEG2_BASE);


However, I  think I may be in some sort of compiler optimization hell.  It seems impossible to retrieve the pointers to the const char*s with pgm_read_word_far...but I can get to the words themselves, which are stored sequentially in PROGMEM_SEG2 and thus seem like one big long string.

I also ran into the perennial issue with the Mega2560 bootloader hanging...the new version of the bootloader hex does seem to work but I wind up with a problem on verification.  It gives me a warning, but the code loads just fine.

If anyone followed all that, thanks much for your valuable time.

With kind regards
Joe
5  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 04, 2013, 12:18:34 pm
Ok.  Progress after many hours of bashing. I promised to report back, and I am.  
Consider this my engineering notebook of sorts.  I'm not trying to instruct you experts, but rather, just recording my experiences. I know I do not have a complete understanding yet but this is as far as I got.  I need more info - particularly on memory sections and addressing. But Maybe this will be valuable to someone.

My application is that I am putting a dictionary of 10k words into an Arduino Mega2560 and accessing it through some means for display on an LCD.  My IDE is (mostly) the Arduino 1.0.3, and I'm primarily using MacOSX v10.6.  What I eventually got to work was on that system.

I also tried compile/link/load sequence from the command line using avr-gcc 4.6.2 and 4.7.2.  I tried the AVR environment on Eclipse and also on Mac XCode.  Results varied in different ways with all those methods.  In all cases I'm using the most version of avrdude (I forget which) that comes with 1.0.3, and also which is obtained when you fetch it on MacOSX with macports or fink.  It's the same version in all cases.

I'm pretty confident that the combination of 4.7.2 and XCodeAVR or EclipseAVR would give me an entirely different experience than the ArduinoIDE.  However I didn't do more than blinky-with-big-global-vars on any of those because I didn't have the time/patience to go back and recast all my Arduino code in native AVR speak.  However I was very successful in compiling and linking to a .hex  that contained flash/global data structures larger than 64k.  I did not attempt to access those,  though.

Things learned:

There is a real 64k boundary for data today.  Note that code is not as limited.  The vector long jump table (which I guess is called a "trampoline table" in AVR speak) consists of 32 bit words and so could easily address much more than the 256k in the Mega.  However, data access is limited by 16-bit pointer addressing, so the largest chunk is 64k.   Now, there is a great exception to this. By concatenating bits from another register (RAMPZ) you can effectively create a 24bit pointer for data, and this is done, partially in some asm code available in multiple libraries out there.  At the moment, on the version of the IDE we have, though you can specify multiple 64k byte clumps, you can't cross that boundary on any one array or data structure.

Future versions of avr-gcc seem to address this.  Particularly, it appears that in 4.7.2 the compiler/linker is happy with larger-than-64k global defs.   And let me say here - it's global defs that are not-changing that we are talking about at all times, because it goes into flash.  If you say"
Code:
volatile int xyz = 123;
 
That gets put into RAM and is globally defined for all your code/ISRs.  This is not what I'm saying.  If I say

Code:
volatile int xyz PROGMEM_blahblah = 123;

The "volatile" piece is not particularly useful, far as I can tell.  By specifying PROGMEM_... you're putting the data into flash, and as such, it has to be globally accessible because you can't put PROGMEM data into local vars. And as it's not changable this would be just as effective and apparently does exactly the same thing:

Code:
int xyz PROGMEM_blahblah = 123;

The problem with getting things to compile when you've got more than 64k in one chunk (and it will compile under certain circumstances) is that the rest of the system has no clue how to handle it because the ptr to the globals is only 16bits and theres nothing upon nothing you can do about that.  So you wind up with linker errors, and errors that show up in other places - in code you had nothing to do with.  This is a sign of badness, and the signal that you need to accept your 64k limitation with happiness and move on.  Because it will be solved at some point.

In the version of avr-gcc available in the Ar1.0.3 (I believe it's 4.3.2) distribution, the compiler balks at "certain" declarations/structures which when initialized reach over 64k.  And this is a key point - most of the problems surface in the "initialization" of defined vars, and not in the definition itself.

By the way, you can happily define/allocate empty vars to your hearts content.  This is a red herring,  though, I have found.  In a lot of the tests run here, and also ones I've tried, you can play various games to get the compiler to NOT optimize away unused/uninitialized space.  But the results are variable.  

First, let me indicate we are talking about data in Flash.  For all intent and purposes, data in Flash may as well be data in a ROM.  Yes, I know there are ways to modify it during runtime, but that goes beyond what I have tried here.

As the flash data is essentially in a ROM, it is not unreasonable to expect that you would know what it was apriori.  That is - you're going to burn it into flash, so you're certain what the data is.  Therefore, all vars/data going into flash are known ahead of time and the compilers/linkers presume as much.  This is an important point, and also one which is the source of a lot of pain.  You can do the following in your code:

Code:
const char abc[]  PROGMEM_blahBlah = "abc"; // PROGMEM_blahblah to be explained

And the abc will become abc[4] after compilation - that is, three chars 'a','b','c' and a trailing null '\0'.  Please note that the compilers presume that declaration/initialization is intended to define a string.  You don't get to come back later and say - hey, there's only 3 chars I didn't mean for it to be a null-terminated string.   Too bad.  The compiler is helping you.   If you want a 3 char array you say:

Code:
const char abc[3]  PROGMEM_blahBlah = 'a','b','c';


and you get that. But in the determination of your memory usage you may be thrown off when the compiler tries to help you by adding another byte. In addition, when you're doing pointer arithmetic, you can't (always) address things in various parts of flash.  For instance, depending on how you organized things this gives you garbage:

Code:
int aa[3] PROGMEM_yak= 1,2,3;
int fetchedInt;
for(int x=0;x,3;x++){
fetchedInt = pgm_read_word_far(&aa[x]);
printf("%d=%d\n",x,fetchedInt);
}
where this will work
Code:
int aa[3] PROGMEM_yak= 1,2,3;
int a,b,c;
a = pgm_read_word(&aa[1]);
b = pgm_read_word(&aa[2]);
c = pgm_read_word(&aa[3]);
printf("a=%d,b=%d,c=%d\n",a,b,c);

Now in my app, I am defining 10,000 words in a way I can access them.  I have tried several methods.  I have tried putting them in a struct like this:

Code:
typedef struct {
         const char a[2];
         const char aardvark[9];
         //... etc 10k words
} words;

const words dictionary PROGMEM_blahblah = {
             {"a"}, {"aardvark"},//etc, 10000 initializers};

And that will compile just with the current version of avr-gcc.   However, it generates an error on versions 4.6.2 and 4.7.2 that say something to the effect of "internal error:  report a bug ..."

Doing this

 
Code:
typedef struct {
         const char a[2];
         const char aardvark[9];
         //... etc 10k words
} words;

const words dictionary PROGMEM_blahblah = {
             .a="a",
             .aardvark="aardvark",
             //etc, 10000 initializers};

Will not compile on the current Arduino IDE but will compile on version 4.6.2 and v4.7.2.

I inevitably settled on a different means - and this isn't entirely debugged.  I'm still having trouble with the flash memory sections, but I did the following:

Code:

const char apple[6] PROGMEM_yadda="apple";
const char bear[5] PROGMEM_yadda="bear";
// etc. etc. etc. 10000 words

const char* dictionary[10000] PROGMEM_yadda+1 ={apple, bear, //etc etc


The idea is that 64k of dictionary data is in a chunk of flash designated by the attribute "PROGMEM_yadda" and an array of pointers to that data is in a chunk of memory called "PROGMEM_yadda+1"  (it may be obvious to most - but please don't try to create code with PROGMEM_yadda... that's just an example )  That compiles and links peachily.  

Now - PROGMEM what I learned about PROGMEM is that as an attribute it specifies putting data into flash.  Using PROGMEM is a multi-step process.  You have to put it into your code, explicitly.  But you also have to change the linker script to understand what you mean by that attribute.  And then retrieving data stored via PROGMEM requires using accessors of the form

Code:
pgm_read_word_far(&myArray[i]);



...continued
6  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 02, 2013, 04:53:45 pm
My problem appears to be this problem:

Quote
..it looks like the version of g++ used by Arduino will fail whenever the global constructors get pushed beyond the 64k limit, because the global constructor table is only 16bits wide and the code uses ijmp to access it...

Found here: http://code.google.com/p/arduino/issues/detail?id=1067

I'm compiling avr-gcc-4.7.2 right now just for grins, and I will try building outside the Arduino IDE environment to see if I can get it to load.

Ah, I had long forgotten the joys of code.
7  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 01, 2013, 04:48:52 pm
Thanks Nick.
Yes, absolutely, a more intelligent data structure would be more compact and faster too.
Of course, even then there is the bald-faced challenge of trying to use up the whole 256k of the Mega....and once solved I'll also use tries and get even more packed in there!!!
Cheers
Joe
8  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 01, 2013, 02:55:27 pm
Ok, an experiments with Lefty's code.   I generated some exhaustive initialization just to see what would happen.

Code:
#include <avr/pgmspace.h>   //To store arrays into flash rather then SRAM
// Simple sketch to create large sketch sizes for testing purposes
/*
  Blink
  Turns on an LED on for one second, then off for one second, repeatedly.
 
  This example code is in the public domain.
 */
 
// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

/*
 Make arraysize = to 1500 for 328P chip, 4000 for 1280P chip?,
 3600 for 644P chip, xxxx for 1284P,  etc.
*/
const int arraysize= 3000;  // value to mostly fill available flash capacity

long myInts0[arraysize] PROGMEM = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,... // up to 2999
long myInts1[arraysize] PROGMEM =  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,... // up to 2999
//...
//...up to
long myInts9[arraysize]PROGMEM={//etc

void setup() {               
  // initialize the digital pin as an output.
  pinMode(led, OUTPUT);
  int i = random(0,arraysize);      // Work around any optimization for constant values
  Serial.print(myInts0[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts1[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts2[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts3[i]);         //  Access some random element so the array can't be optimized away.
}

// the loop routine runs over and over again forever:
void loop() {
 
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
 
}


What I've noticed is that if the sketch size goes over 128K I start getting errors of the form:

warning: internal error: out of range error...

That is, if I add a an additional array of [arraysize] longs where arraysize = 3000, bringing the number up to 11 arrays, each of 3000 long elements = 4*11*3000 = 132K bytes (then add the rest of the sketch )  I start getting that out of range error called on various libs.  For instance the first place it shows up is as an out of range error on the do_random func random.o.   If I comment out the call to random, it shows up in Hardware Serial.

If I stick to 10 initialized arrays the sketch size is 124,996 ( = 4*10*3000 + rest of sketch).  No problems compiling/linking.
 

Now interestingly.  suppose instead of long int I initialize each of the myInts array to 3000 4 byte null terminated strings thus:

Code:
char* myInts9[arraysize] PROGMEM ={"abc\0","abc\0","abc\0",//..etc for 3000 initializers

For 10 initialized arrays of 3000 four-byte-strings the IDE reports a sketch size of 64,646 bytes out of a 258,048 byte maximum.  The same sketch initialized to longs is reported as 124,996.

For 20 initialized arrays of 3000 four-byte strings the IDE reports

arduino-1.0.3\hardware\arduino\cores\arduino/main.cpp:11: warning: internal error: out of range error

Binary sketch size: 136,682 bytes (of a 258,048 byte maximum)

If I take out 1 array I get no errors and
Binary sketch size: 130,646 bytes (of a 258,048 byte maximum)


I have not yet tried to load and run these sketches.

Cheers,
Joe
9  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 01, 2013, 01:00:52 pm
Though slightly inconvenient that sounds vastly rational.  In any case the dictionary I am trying to store is a static resource, so getting it set up correctly a-priori is no problem.  Once it is set up, I won't change it.  Hopefully in the future this will only get easier.
 I will try it.
Joe
10  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 01, 2013, 12:37:16 pm
Thanks for the code, Pyro.
That seems to definitely work to access the data in Flash. 

The issue I seem to be facing from the outset happens where the declaration is:

prog_char abc[] = "hello world";


In my code, I am trying to create 10,000 prog_char var[]s.   The size of all the strings in 10,000 declarations combined is 75k.  There would have to be at least 17 bits of addressing to accomodate that.  Fortunately, it appears from literally all of the documentation that it is possible to do this on the ATMega2560 chip set through the use of the RAMPZ register.

However, when I try to compile/link the code that contains the 10,000 lines of declarations, each of which is unique and looks like   -  prog_char foo[] = "bar"; -
the compiler/linker stage balks in the Arduino IDE.  The error I get is that those declarations are too large to fit in 2 bytes of addressing.

Well, sure.  It's just math and we all know you can't address 75K of data with only 16 bits.  But there should be flags set in the appropriate places (like --relax in the linker) and defines for the compiler that make sure the addressing is set up correctly.

The issue I'm having is getting the data into the high memory in the first place.   I'm absolutely certain that once it is there, we can use various functionality suggested kindly here to read it.

Thanks so much for putting up with my blather.

Joe
11  Using Arduino / Programming Questions / Re: Just a newby asking the 64k question again - Arduino Mega2560 on: February 01, 2013, 01:25:01 am
Hi and thank you all for the suggestions.

The suggestion to look into "morepgmspace.h" led to the AVR C runtime lib website and Carlos's code.  It looks very instructive, though it is mostly runtime stuff (well, sure, of course).  Now I don't know enough about this platform to understand if perhaps that code suggests you can load the FLASH dynamically - I thot that wasn't possible.  Or perhaps the way I should read it is that I need to somehow write my declaration code so it looks like it's being dynamically written, but actually, it's being compiled and linked and uploaded like any normal sketch.

There is actually a bug fix update/change to pgmspace that was uploaded as recently as a few days ago.  In that version, as opposed to the one in the 1.0.3 distribution, they extern a whole bunch of long address functions.  They also deprecate a whole load of types.

Hemmerling's page says that for chips that have over 64k of flash - that the normal 16bit pointer scheme (plus RAMPZ, etc) allows addressing to 128k in 2 64k chunks.

I did indeed try to split my

const char* array[] {};

into two separate chunks, though it still thinks I'm trying to address flash beyond 64k with only the 16bit ptr, and the assembler balks.

I'm wondering if there isn't a define somewhere that I'm not taking advantage of.  For instance - could it be that selecting "Mega2560 or Mega ADK" somehow isn't getting the right defines set somewhere so it isn't taking advantage of RAMPZ, or perhaps I'm just declaring stuff in a way that's braindead.

Most likely, I'm doing something braindead, and I expect there's a simple newbie mistake in my process that once I figure out will have me slapping my forehead into the wall...    Discussion of how to address and read flash over 64k is everywhere.  But how to write it at compile/link/load time is not.

That makes me think it should just happen easily, and I'm missing something obvious (as usual).

Thanks for the code snippets.  I will try all avenues and report back if I have a breakthrough.

Best
Joe
12  Using Arduino / Programming Questions / Just a newby asking the 64k question again - Arduino Mega2560 on: January 31, 2013, 06:12:35 pm
Hi all and thanks in advance,

For the past week I have been scouring the web and this site with great interest in trying to answer the perennial question - can you store an array in PROGMEM that is larger than 64k bytes on the Mega2560?

My application is that I want to store a dictionary of 10,000 words in PROGMEM statically through compilation in the usual way prescribed here and in other places:

Code:
#include <avr/pgmspace.h>
const char* a_dict PROGMEM = "a";
const char* aa_dict PROGMEM ="aa";
const char* aaa_dict PROGMEM ="aaa";
const char* aaron_dict  PROGMEM ="aaron";
const char* ab_dict PROGMEM ="ab";
const char* abandoned_dict PROGMEM ="abandoned";
const char* abc_dict PROGMEM ="abc";
//.
//.
//.
 //etc  10,000 lines of words.  The total size of all the characters+null termination bytes in all the words is 75.5k. ...then...

const char* dictionary[] PROGMEM = {a_dict, aa_dict,aaa_dict,aaron_dict,ab_dict,abandoned_dict,abc_dict // 10000 char* vars worth
};
//then some way to access the dictionary yet to be decided
void setup(){}

void loop(){
char* x; //a place holder for a recovered string
int index; //some sort of index
//something like strcpy_PF(x, (char*)pgm_read_word_far(&(dictionary[index])));
// yadda yadda, do something with x
}


The problem I run into is the thing will not compile.  I get an assembler error saying that "Error: value of 68731 is too large to fit into 2 bytes at xyz"  which of course is true for any number bigger than 2^16

So I'm thinking there must be some secret incantation to get it to be understood that I need addressing to somehow be far/long/etc...somehow at least 1 bit more than 16 bits of addressing.  It looks like by reading the various .h files and .cpp files in the avr libs that there is a register called RAMPZ that is used in the extension of the addressing to allow addressing more than 64K of flash.  But how to get that to happen?   

In all my reading I find a lot of info on how to access data placed above the 64k boundary - but nobody says how to get the compiler to put it up there in the first place.  Is there a parameter?  Is there a typedef I'm missing?   Or - does the compiler just simply do it for you when you set things up correctly.

Should I divide up my dictionary of words into 2 pieces, each less than 64k?  Then perhaps I could do something like __attribute__((section(".someothersection"))); to get it to go in some other place?

Any guidance would be muchly appreciated.

Quite humbly,
Joe

Pages: [1]