Arduino Forum

Using Arduino => Programming Questions => Topic started by: iceowl on Feb 01, 2013, 12:12 am

Title: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 12:12 am
Hi all and thanks in advance,

For the past week I have been scouring the web and this site with great interest in trying to answer the perennial question - can you store an array in PROGMEM that is larger than 64k bytes on the Mega2560?

My application is that I want to store a dictionary of 10,000 words in PROGMEM statically through compilation in the usual way prescribed here and in other places:

Code: [Select]

#include <avr/pgmspace.h>
const char* a_dict PROGMEM = "a";
const char* aa_dict PROGMEM ="aa";
const char* aaa_dict PROGMEM ="aaa";
const char* aaron_dict  PROGMEM ="aaron";
const char* ab_dict PROGMEM ="ab";
const char* abandoned_dict PROGMEM ="abandoned";
const char* abc_dict PROGMEM ="abc";
//.
//.
//.
//etc  10,000 lines of words.  The total size of all the characters+null termination bytes in all the words is 75.5k. ...then...

const char* dictionary[] PROGMEM = {a_dict, aa_dict,aaa_dict,aaron_dict,ab_dict,abandoned_dict,abc_dict // 10000 char* vars worth
};
//then some way to access the dictionary yet to be decided
void setup(){}

void loop(){
char* x; //a place holder for a recovered string
int index; //some sort of index
//something like strcpy_PF(x, (char*)pgm_read_word_far(&(dictionary[index])));
// yadda yadda, do something with x
}



The problem I run into is the thing will not compile.  I get an assembler error saying that "Error: value of 68731 is too large to fit into 2 bytes at xyz"  which of course is true for any number bigger than 2^16

So I'm thinking there must be some secret incantation to get it to be understood that I need addressing to somehow be far/long/etc...somehow at least 1 bit more than 16 bits of addressing.  It looks like by reading the various .h files and .cpp files in the avr libs that there is a register called RAMPZ that is used in the extension of the addressing to allow addressing more than 64K of flash.  But how to get that to happen?   

In all my reading I find a lot of info on how to access data placed above the 64k boundary - but nobody says how to get the compiler to put it up there in the first place.  Is there a parameter?  Is there a typedef I'm missing?   Or - does the compiler just simply do it for you when you set things up correctly.

Should I divide up my dictionary of words into 2 pieces, each less than 64k?  Then perhaps I could do something like __attribute__((section(".someothersection"))); to get it to go in some other place?

Any guidance would be muchly appreciated.

Quite humbly,
Joe

Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: michinyon on Feb 01, 2013, 12:41 am
Do what the phone companies used to do.  Divide your dictionary into two parts  A-L  and M-Z

I don't think there is any "secret incantation" that changes pointer addressing from 2 to 4 bytes
for the Arduino.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: lloyddean on Feb 01, 2013, 12:49 am
I don't believe this 256K FLASH memory can be used to store and retrieve data and is meant to be used for executable code only.

EDIT:

An near Arduino compatible such as the chipKIT MAX 32 may be more useful for your application.

Microchip® PIC32MX795F512 processor
80 Mhz 32-bit MIPS
512K Flash, 128K RAM

<http://www.digilentinc.com/Products/Detail.cfm?NavPath=2,892,894&Prod=CHIPKIT-MAX32>
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 01, 2013, 12:57 am
I haven't used it yet, but plan on it as my app is quite large.
I found people mentioning a certain library, hunt around for:

Quote
Carlos Lamas' morepgmspace.h


If you have some luck, report back.

Quote
I don't believe this 256K FLASH memory can be used to store and retrieve data and is meant to be used for executable code only.


People can and do use the higher end of flash for PGM data, keeping the program code in the lower region of flash is far more efficient than  having PROGMEM data push it into the higher region requiring an extra level of indirection.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: lloyddean on Feb 01, 2013, 12:59 am
I learned alonggggg time ago saying something can't be done brings those with useful information out of the woodwork ...
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: nickgammon on Feb 01, 2013, 03:06 am
Since you are only slightly over the limit you might want to try storing the dictionary as a Trie.

http://en.wikipedia.org/wiki/Trie
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: retrolefty on Feb 01, 2013, 06:00 am
Based on my experiments with using large arrays in FLASH, I found I could not get them to work correctly past the 64KB mark on even a mega1280 chip. Was told there is a bug in the older version of the gcc compiler that the arduino distrubution uses, but that newer versions have fixed, but I don't know that for a fact, just what's been posted around here.

Anyway here is a simple testing sketch you are free to play with. By changing one constant, arraysize, you can try and create a sketch of any size you wish. I was able to fill up a 328P and a 644P chip to near capacity, no problem. But cannot get sketch to run properly if the compiled sketch crosses the 64KB mark on the 1280 chip, and I don't have a 2560 board to play with. Symptom on the 'too large' sketches is it compiles without error and appears to upload completely but the blink portion of the sketch never executes.

Code: [Select]

#include <avr/pgmspace.h>   //To store arrays into flash rather then SRAM
// Simple sketch to create large sketch sizes for testing purposes
/*
 Blink
 Turns on an LED on for one second, then off for one second, repeatedly.

 This example code is in the public domain.
*/

// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

/*
Make arraysize = to 1500 for 328P chip, 4000 for 1280P chip?,
3600 for 644P chip, xxxx for 1284P,  etc.
*/
const int arraysize= 1500;  // value to mostly fill available flash capacity

long myInts0[arraysize] PROGMEM = {};  //Store initialized array into flash memory
long myInts1[arraysize] PROGMEM = {};
long myInts2[arraysize] PROGMEM = {};
long myInts3[arraysize] PROGMEM = {};

// the setup routine runs once when you press reset:
void setup() {                
 // initialize the digital pin as an output.
 pinMode(led, OUTPUT);
 int i = random(0,arraysize);      // Work around any optimization for constant values
 Serial.print(myInts0[i]);         //  Access some random element so the array can't be optimized away.
 Serial.print(myInts1[i]);         //  Access some random element so the array can't be optimized away.
 Serial.print(myInts2[i]);         //  Access some random element so the array can't be optimized away.
 Serial.print(myInts3[i]);         //  Access some random element so the array can't be optimized away.
}

// the loop routine runs over and over again forever:
void loop() {
 
 digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
 delay(1000);               // wait for a second
 digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
 delay(1000);               // wait for a second
}
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 07:25 am
Hi and thank you all for the suggestions.

The suggestion to look into "morepgmspace.h" led to the AVR C runtime lib website and Carlos's code.  It looks very instructive, though it is mostly runtime stuff (well, sure, of course).  Now I don't know enough about this platform to understand if perhaps that code suggests you can load the FLASH dynamically - I thot that wasn't possible.  Or perhaps the way I should read it is that I need to somehow write my declaration code so it looks like it's being dynamically written, but actually, it's being compiled and linked and uploaded like any normal sketch.

There is actually a bug fix update/change to pgmspace that was uploaded as recently as a few days ago.  In that version, as opposed to the one in the 1.0.3 distribution, they extern a whole bunch of long address functions.  They also deprecate a whole load of types.

Hemmerling's page says that for chips that have over 64k of flash - that the normal 16bit pointer scheme (plus RAMPZ, etc) allows addressing to 128k in 2 64k chunks.

I did indeed try to split my

const char* array[] {};

into two separate chunks, though it still thinks I'm trying to address flash beyond 64k with only the 16bit ptr, and the assembler balks.

I'm wondering if there isn't a define somewhere that I'm not taking advantage of.  For instance - could it be that selecting "Mega2560 or Mega ADK" somehow isn't getting the right defines set somewhere so it isn't taking advantage of RAMPZ, or perhaps I'm just declaring stuff in a way that's braindead.

Most likely, I'm doing something braindead, and I expect there's a simple newbie mistake in my process that once I figure out will have me slapping my forehead into the wall...    Discussion of how to address and read flash over 64k is everywhere.  But how to write it at compile/link/load time is not.

That makes me think it should just happen easily, and I'm missing something obvious (as usual).

Thanks for the code snippets.  I will try all avenues and report back if I have a breakthrough.

Best
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 01, 2013, 01:18 pm
This looks like a winner.

pgm_read_byte is functionally, pgm_read_byte_near.

What we need is 'pgm_read_byte_far' which accepts a 32-bit pointer allowing us to access the second half of flash. ( 64K words = 128K bytes ).

This macro below translates a PGM (16 bit ) address to a far PGM ( 24-bit, stored as a 32-bit ) address.

So basically, use GET_FAR_ADDRESS to 'get' the address, then use pgm_read_byte_far to read the data.

All the normal *_P functions only support near addresses, you'll have to provide your own custom overloads if needed.

People have had success with this, so hope it helps.

Code: [Select]
#include <inttypes.h>
#include <avr/io.h>
#include <avr/pgmspace.h>
...
prog_char MyString[] = "Hello World";
...

#define GET_FAR_ADDRESS(var)                          \
({                                                    \
   uint_farptr_t tmp;                                \
                                                     \
   __asm__ __volatile__(                             \
                                                     \
           "ldi    %A0, lo8(%1)"           "\n\t"    \
           "ldi    %B0, hi8(%1)"           "\n\t"    \
           "ldi    %C0, hh8(%1)"           "\n\t"    \
           "clr    %D0"                    "\n\t"    \
       :                                             \
           "=d" (tmp)                                \
       :                                             \
           "p"  (&(var))                             \
   );                                                \
   tmp;                                              \
})

void UART0_puts_P(uint_farptr_t str)
//---------------------------------------------------
// send a ProgMem string to UART0 Transmit-buffer
//---------------------------------------------------
{
 u08 c= pgm_read_byte_far(str);
 while (c)
 {
   UART0_putc(c);
   c=pgm_read_byte_far(++str);    
 }
}
...
int main(void)
{
 ...
 UART0_puts_P(GET_FAR_ADDRESS(MyString));
 while(1);
}
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: retrolefty on Feb 01, 2013, 04:23 pm

This looks like a winner.

pgm_read_byte is functionally, pgm_read_byte_near.

What we need is 'pgm_read_byte_far' which accepts a 32-bit pointer allowing us to access the second half of flash. ( 64K words = 128K bytes ).

This macro below translates a PGM (16 bit ) address to a far PGM ( 24-bit, stored as a 32-bit ) address.

So basically, use GET_FAR_ADDRESS to 'get' the address, then use pgm_read_byte_far to read the data.

All the normal *_P functions only support near addresses, you'll have to provide your own custom overloads if needed.

People have had success with this, so hope it helps.

Code: [Select]
#include <inttypes.h>
#include <avr/io.h>
#include <avr/pgmspace.h>
...
prog_char MyString[] = "Hello World";
...

#define GET_FAR_ADDRESS(var)                          \
({                                                    \
   uint_farptr_t tmp;                                \
                                                     \
   __asm__ __volatile__(                             \
                                                     \
           "ldi    %A0, lo8(%1)"           "\n\t"    \
           "ldi    %B0, hi8(%1)"           "\n\t"    \
           "ldi    %C0, hh8(%1)"           "\n\t"    \
           "clr    %D0"                    "\n\t"    \
       :                                             \
           "=d" (tmp)                                \
       :                                             \
           "p"  (&(var))                             \
   );                                                \
   tmp;                                              \
})

void UART0_puts_P(uint_farptr_t str)
//---------------------------------------------------
// send a ProgMem string to UART0 Transmit-buffer
//---------------------------------------------------
{
 u08 c= pgm_read_byte_far(str);
 while (c)
 {
   UART0_putc(c);
   c=pgm_read_byte_far(++str);    
 }
}
...
int main(void)
{
 ...
 UART0_puts_P(GET_FAR_ADDRESS(MyString));
 while(1);
}



So how can this 'solution' be implemented in context of writing and uploading sketches in the Arduino IDE?

Lefty
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 06:37 pm
Thanks for the code, Pyro.
That seems to definitely work to access the data in Flash. 

The issue I seem to be facing from the outset happens where the declaration is:

prog_char abc[] = "hello world";


In my code, I am trying to create 10,000 prog_char var[]s.   The size of all the strings in 10,000 declarations combined is 75k.  There would have to be at least 17 bits of addressing to accomodate that.  Fortunately, it appears from literally all of the documentation that it is possible to do this on the ATMega2560 chip set through the use of the RAMPZ register.

However, when I try to compile/link the code that contains the 10,000 lines of declarations, each of which is unique and looks like   -  prog_char foo[] = "bar"; -
the compiler/linker stage balks in the Arduino IDE.  The error I get is that those declarations are too large to fit in 2 bytes of addressing.

Well, sure.  It's just math and we all know you can't address 75K of data with only 16 bits.  But there should be flags set in the appropriate places (like --relax in the linker) and defines for the compiler that make sure the addressing is set up correctly.

The issue I'm having is getting the data into the high memory in the first place.   I'm absolutely certain that once it is there, we can use various functionality suggested kindly here to read it.

Thanks so much for putting up with my blather.

Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: krupski on Feb 01, 2013, 06:55 pm


In all my reading I find a lot of info on how to access data placed above the 64k boundary - but nobody says how to get the compiler to put it up there in the first place.  Is there a parameter?  Is there a typedef I'm missing?   Or - does the compiler just simply do it for you when you set things up correctly.


I've run into this problem and I'm not SURE, but it seems like it happens whenever a PROGMEM access has to CROSS a 64K boundary. I wasn't able to find out for sure because to test it I would need "offending" code and "offending" code won't compile......  :)

What I've done was to, by trial and error, find where my stuff was crossing a 64K boundary (I think), then use a dummy variable to use up the rest of the first 64K so that real variables were in the next 64K, and so on...

Sadly, I'm not even sure if I'm on the right track... all I know it I got it to work this way.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 07:00 pm
Though slightly inconvenient that sounds vastly rational.  In any case the dictionary I am trying to store is a static resource, so getting it set up correctly a-priori is no problem.  Once it is set up, I won't change it.  Hopefully in the future this will only get easier.
I will try it.
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 08:55 pm
Ok, an experiments with Lefty's code.   I generated some exhaustive initialization just to see what would happen.

Code: [Select]

#include <avr/pgmspace.h>   //To store arrays into flash rather then SRAM
// Simple sketch to create large sketch sizes for testing purposes
/*
  Blink
  Turns on an LED on for one second, then off for one second, repeatedly.

  This example code is in the public domain.
*/

// Pin 13 has an LED connected on most Arduino boards.
// give it a name:
int led = 13;

/*
Make arraysize = to 1500 for 328P chip, 4000 for 1280P chip?,
3600 for 644P chip, xxxx for 1284P,  etc.
*/
const int arraysize= 3000;  // value to mostly fill available flash capacity

long myInts0[arraysize] PROGMEM = {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,... // up to 2999
long myInts1[arraysize] PROGMEM =  {0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,
30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,... // up to 2999
//...
//...up to
long myInts9[arraysize]PROGMEM={//etc

void setup() {               
  // initialize the digital pin as an output.
  pinMode(led, OUTPUT);
  int i = random(0,arraysize);      // Work around any optimization for constant values
  Serial.print(myInts0[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts1[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts2[i]);         //  Access some random element so the array can't be optimized away.
  Serial.print(myInts3[i]);         //  Access some random element so the array can't be optimized away.
}

// the loop routine runs over and over again forever:
void loop() {
 
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
 
}



What I've noticed is that if the sketch size goes over 128K I start getting errors of the form:

warning: internal error: out of range error...

That is, if I add a an additional array of [arraysize] longs where arraysize = 3000, bringing the number up to 11 arrays, each of 3000 long elements = 4*11*3000 = 132K bytes (then add the rest of the sketch )  I start getting that out of range error called on various libs.  For instance the first place it shows up is as an out of range error on the do_random func random.o.   If I comment out the call to random, it shows up in Hardware Serial.

If I stick to 10 initialized arrays the sketch size is 124,996 ( = 4*10*3000 + rest of sketch).  No problems compiling/linking.


Now interestingly.  suppose instead of long int I initialize each of the myInts array to 3000 4 byte null terminated strings thus:

Code: [Select]

char* myInts9[arraysize] PROGMEM ={"abc\0","abc\0","abc\0",//..etc for 3000 initializers


For 10 initialized arrays of 3000 four-byte-strings the IDE reports a sketch size of 64,646 bytes out of a 258,048 byte maximum.  The same sketch initialized to longs is reported as 124,996.

For 20 initialized arrays of 3000 four-byte strings the IDE reports

arduino-1.0.3\hardware\arduino\cores\arduino/main.cpp:11: warning: internal error: out of range error

Binary sketch size: 136,682 bytes (of a 258,048 byte maximum)

If I take out 1 array I get no errors and
Binary sketch size: 130,646 bytes (of a 258,048 byte maximum)


I have not yet tried to load and run these sketches.

Cheers,
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: nickgammon on Feb 01, 2013, 09:53 pm
It doesn't totally surprise me you are having these problems. My experience has been that the Mega side of the platform, being used less, hasn't received as much attention (eg. bootloaders that don't handle the watchdog timer).

Then people compiling large arrays (in itself perfectly sensible) are probably in a minority too.

It would be interesting if, when you solve this, we document it so others can benefit from it.

Meanwhile you could always consider my suggestion of trying to store your dictionary more compactly and make the problem go away. :)
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 01, 2013, 10:48 pm
Thanks Nick.
Yes, absolutely, a more intelligent data structure would be more compact and faster too.
Of course, even then there is the bald-faced challenge of trying to use up the whole 256k of the Mega....and once solved I'll also use tries and get even more packed in there!!!
Cheers
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 02, 2013, 12:14 am
Quote
So how can this 'solution' be implemented in context of writing and uploading sketches in the Arduino IDE?

Lefty


Copy 'n' paste.


You cannot address huge variables even in the < 64k boundary.
Here is a program using every single byte on the mega using progmem, it is possible, and GET_FAR_ADDRESS will help you read the data.

Code: [Select]
#define nothing

template< uint64_t C, typename T >
 struct LargeStruct{
   T Data;
   LargeStruct< C - 1, T > Next;
};
template< typename T > struct LargeStruct< 0, T >{ };

typedef LargeStruct< 80, uint64_t > Container; //640 bytes

PROGMEM LargeStruct< 50, Container > l_Struct;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct1;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct2;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct3;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct4;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct5;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct6;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct7;   //32k

PROGMEM LargeStruct< 431, uint16_t > l_Struct8; //862 bytes
void setup()
 {
   volatile int i = ( int ) &l_Struct;
   volatile int i1 = ( int ) &l_Struct1;
   volatile int i2 = ( int ) &l_Struct2;
   volatile int i3 = ( int ) &l_Struct3;
   volatile int i4 = ( int ) &l_Struct4;
   volatile int i5 = ( int ) &l_Struct5;
   volatile int i6 = ( int ) &l_Struct6;
   volatile int i7 = ( int ) &l_Struct7;    
   volatile int i8 = ( int ) &l_Struct8;  
 }

void loop(){}
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: retrolefty on Feb 02, 2013, 01:47 am

Quote
So how can this 'solution' be implemented in context of writing and uploading sketches in the Arduino IDE?

Lefty


Copy 'n' paste.


You cannot address huge variables even in the < 64k boundary.
Here is a program using every single byte on the mega using progmem, it is possible, and GET_FAR_ADDRESS will help you read the data.

Code: [Select]
#define nothing

template< uint64_t C, typename T >
 struct LargeStruct{
   T Data;
   LargeStruct< C - 1, T > Next;
};
template< typename T > struct LargeStruct< 0, T >{ };

typedef LargeStruct< 80, uint64_t > Container; //640 bytes

PROGMEM LargeStruct< 50, Container > l_Struct;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct1;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct2;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct3;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct4;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct5;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct6;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct7;   //32k

PROGMEM LargeStruct< 431, uint16_t > l_Struct8; //862 bytes
void setup()
 {
   volatile int i = ( int ) &l_Struct;
   volatile int i1 = ( int ) &l_Struct1;
   volatile int i2 = ( int ) &l_Struct2;
   volatile int i3 = ( int ) &l_Struct3;
   volatile int i4 = ( int ) &l_Struct4;
   volatile int i5 = ( int ) &l_Struct5;
   volatile int i6 = ( int ) &l_Struct6;
   volatile int i7 = ( int ) &l_Struct7;    
   volatile int i8 = ( int ) &l_Struct8;  
 }

void loop(){}



So I added a blink function to your example sketch after removing some of the 'structure stuff' so as to fit in a 1280 chip and uploaded to a mega.

No compile errors, compile size 98,726 of 130,048 maximum, upload proceeds with no errors, but certainly takes a while. When done no blinking led13? Why does the sketch not run? That is the same symptom I was seeing in my code example posted earlier, I could create sketches of desirable size but after a certain size the blink in loop() doesn't execute?

Code: [Select]

#define nothing

template< uint64_t C, typename T >
 struct LargeStruct{
   T Data;
   LargeStruct< C - 1, T > Next;
};
template< typename T > struct LargeStruct< 0, T >{ };

typedef LargeStruct< 80, uint64_t > Container; //640 bytes

PROGMEM LargeStruct< 50, Container > l_Struct;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct1;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct2;   //32k
/*PROGMEM LargeStruct< 50, Container > l_Struct3;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct4;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct5;  //32k
PROGMEM LargeStruct< 50, Container > l_Struct6;   //32k
PROGMEM LargeStruct< 50, Container > l_Struct7;   //32k
*/
PROGMEM LargeStruct< 431, uint16_t > l_Struct8; //862 bytes
int led = 13;
void setup()
 {
   pinMode(led, OUTPUT);
   volatile int i = ( int ) &l_Struct;
   volatile int i1 = ( int ) &l_Struct1;
   volatile int i2 = ( int ) &l_Struct2;
/*   volatile int i3 = ( int ) &l_Struct3;
   volatile int i4 = ( int ) &l_Struct4;
   volatile int i5 = ( int ) &l_Struct5;
   volatile int i6 = ( int ) &l_Struct6;
   volatile int i7 = ( int ) &l_Struct7;    
   volatile int i8 = ( int ) &l_Struct8;
 */  
 }

void loop(){

digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
 delay(1000);               // wait for a second
 digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
 delay(1000);               // wait for a second
}



Lefty
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 02, 2013, 11:38 am
Good find, I didn't try anything like that. I previously wasn't able to answer the questions as that sketch is a bit obscure, but as I need pgm functionality, I've done a little investigating to work out what is happening.

Firstly, the structures placed into progmem are considered first, before the functions. So as a consequence, the function's code ( main, loop, pinMode, etc... ) is placed in a region not accessible by conventional 16-bit pointers. Therefore to call this code you have to jump to the trampoline, which in turn contains a jump to your code.

From what I was able to ingest, functions that exist in the >64K word boundary automatically have an entry in a thing called a 'trampoline'. It is a table of jumps to locations in the higher memory, allowing the entire memory range to be used.

Also as I understand it, if main is in high memory, it will still be called via the trampoline. The problem with this sketch is, not that the 'far' code isn't being called.
Its just that the arduino libraries do not expect their PGM data to be out of range.

There are a number of ways to get things working. For instance you could update the core to use far pointers where necessary, or the easiest quick fix is to place the structure data after the functions so the core can have full use of the lower address range, however there is an overhead to using high range access, critical stuff should remain in the lower section.

This macro will place data after other sections. Found here (http://arduino.cc/forum/index.php?topic=134649.0)
Code: [Select]
#define PROGMEM_FAR  __attribute__((section(".fini7")))

Code: [Select]
#define nothing

template< uint64_t C, typename T >
  struct LargeStruct{
    T Data;
    LargeStruct< C - 1, T > Next;
};
template< typename T > struct LargeStruct< 0, T >{ };

typedef LargeStruct< 80, uint64_t > Container; //640 bytes

#define PROGMEM_FAR  __attribute__((section(".fini7")))

PROGMEM_FAR LargeStruct< 50, Container > l_Struct;  //32k
PROGMEM_FAR LargeStruct< 50, Container > l_Struct1;   //32k
PROGMEM_FAR LargeStruct< 50, Container > l_Struct2;   //32k
PROGMEM_FAR LargeStruct< 431, uint16_t > l_Struct8; //862 bytes

int led = 13;

void setup()
  {
    pinMode(led, OUTPUT);
    volatile int i = ( int ) &l_Struct;
    volatile int i1 = ( int ) &l_Struct1;
    volatile int i2 = ( int ) &l_Struct2;
  }

void loop(){

digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(1000);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(1000);               // wait for a second
}

Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 02, 2013, 10:53 pm
My problem appears to be this problem:

Quote
..it looks like the version of g++ used by Arduino will fail whenever the global constructors get pushed beyond the 64k limit, because the global constructor table is only 16bits wide and the code uses ijmp to access it...


Found here: http://code.google.com/p/arduino/issues/detail?id=1067

I'm compiling avr-gcc-4.7.2 right now just for grins, and I will try building outside the Arduino IDE environment to see if I can get it to load.

Ah, I had long forgotten the joys of code.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 02, 2013, 11:27 pm
Read the post I wrote above, the answer is in there.

Move your pgm data after the code, then the constructor table starts in the lower address memory.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 04, 2013, 06:18 pm
Ok.  Progress after many hours of bashing. I promised to report back, and I am.  
Consider this my engineering notebook of sorts.  I'm not trying to instruct you experts, but rather, just recording my experiences. I know I do not have a complete understanding yet but this is as far as I got.  I need more info - particularly on memory sections and addressing. But Maybe this will be valuable to someone.

My application is that I am putting a dictionary of 10k words into an Arduino Mega2560 and accessing it through some means for display on an LCD.  My IDE is (mostly) the Arduino 1.0.3, and I'm primarily using MacOSX v10.6.  What I eventually got to work was on that system.

I also tried compile/link/load sequence from the command line using avr-gcc 4.6.2 and 4.7.2.  I tried the AVR environment on Eclipse and also on Mac XCode.  Results varied in different ways with all those methods.  In all cases I'm using the most version of avrdude (I forget which) that comes with 1.0.3, and also which is obtained when you fetch it on MacOSX with macports or fink.  It's the same version in all cases.

I'm pretty confident that the combination of 4.7.2 and XCodeAVR or EclipseAVR would give me an entirely different experience than the ArduinoIDE.  However I didn't do more than blinky-with-big-global-vars on any of those because I didn't have the time/patience to go back and recast all my Arduino code in native AVR speak.  However I was very successful in compiling and linking to a .hex  that contained flash/global data structures larger than 64k.  I did not attempt to access those,  though.

Things learned:

There is a real 64k boundary for data today.  Note that code is not as limited.  The vector long jump table (which I guess is called a "trampoline table" in AVR speak) consists of 32 bit words and so could easily address much more than the 256k in the Mega.  However, data access is limited by 16-bit pointer addressing, so the largest chunk is 64k.   Now, there is a great exception to this. By concatenating bits from another register (RAMPZ) you can effectively create a 24bit pointer for data, and this is done, partially in some asm code available in multiple libraries out there.  At the moment, on the version of the IDE we have, though you can specify multiple 64k byte clumps, you can't cross that boundary on any one array or data structure.

Future versions of avr-gcc seem to address this.  Particularly, it appears that in 4.7.2 the compiler/linker is happy with larger-than-64k global defs.   And let me say here - it's global defs that are not-changing that we are talking about at all times, because it goes into flash.  If you say"
Code: [Select]
volatile int xyz = 123;  
That gets put into RAM and is globally defined for all your code/ISRs.  This is not what I'm saying.  If I say

Code: [Select]
volatile int xyz PROGMEM_blahblah = 123;

The "volatile" piece is not particularly useful, far as I can tell.  By specifying PROGMEM_... you're putting the data into flash, and as such, it has to be globally accessible because you can't put PROGMEM data into local vars. And as it's not changable this would be just as effective and apparently does exactly the same thing:

Code: [Select]
int xyz PROGMEM_blahblah = 123;

The problem with getting things to compile when you've got more than 64k in one chunk (and it will compile under certain circumstances) is that the rest of the system has no clue how to handle it because the ptr to the globals is only 16bits and theres nothing upon nothing you can do about that.  So you wind up with linker errors, and errors that show up in other places - in code you had nothing to do with.  This is a sign of badness, and the signal that you need to accept your 64k limitation with happiness and move on.  Because it will be solved at some point.

In the version of avr-gcc available in the Ar1.0.3 (I believe it's 4.3.2) distribution, the compiler balks at "certain" declarations/structures which when initialized reach over 64k.  And this is a key point - most of the problems surface in the "initialization" of defined vars, and not in the definition itself.

By the way, you can happily define/allocate empty vars to your hearts content.  This is a red herring,  though, I have found.  In a lot of the tests run here, and also ones I've tried, you can play various games to get the compiler to NOT optimize away unused/uninitialized space.  But the results are variable.  

First, let me indicate we are talking about data in Flash.  For all intent and purposes, data in Flash may as well be data in a ROM.  Yes, I know there are ways to modify it during runtime, but that goes beyond what I have tried here.

As the flash data is essentially in a ROM, it is not unreasonable to expect that you would know what it was apriori.  That is - you're going to burn it into flash, so you're certain what the data is.  Therefore, all vars/data going into flash are known ahead of time and the compilers/linkers presume as much.  This is an important point, and also one which is the source of a lot of pain.  You can do the following in your code:

Code: [Select]
const char abc[]  PROGMEM_blahBlah = "abc"; // PROGMEM_blahblah to be explained

And the abc will become abc[4] after compilation - that is, three chars 'a','b','c' and a trailing null '\0'.  Please note that the compilers presume that declaration/initialization is intended to define a string.  You don't get to come back later and say - hey, there's only 3 chars I didn't mean for it to be a null-terminated string.   Too bad.  The compiler is helping you.   If you want a 3 char array you say:

Code: [Select]
const char abc[3]  PROGMEM_blahBlah = 'a','b','c';

and you get that. But in the determination of your memory usage you may be thrown off when the compiler tries to help you by adding another byte. In addition, when you're doing pointer arithmetic, you can't (always) address things in various parts of flash.  For instance, depending on how you organized things this gives you garbage:

Code: [Select]
int aa[3] PROGMEM_yak= 1,2,3;
int fetchedInt;
for(int x=0;x,3;x++){
fetchedInt = pgm_read_word_far(&aa[x]);
printf("%d=%d\n",x,fetchedInt);
}

where this will work
Code: [Select]
int aa[3] PROGMEM_yak= 1,2,3;
int a,b,c;
a = pgm_read_word(&aa[1]);
b = pgm_read_word(&aa[2]);
c = pgm_read_word(&aa[3]);
printf("a=%d,b=%d,c=%d\n",a,b,c);


Now in my app, I am defining 10,000 words in a way I can access them.  I have tried several methods.  I have tried putting them in a struct like this:

Code: [Select]
typedef struct {
        const char a[2];
        const char aardvark[9];
        //... etc 10k words
} words;

const words dictionary PROGMEM_blahblah = {
            {"a"}, {"aardvark"},//etc, 10000 initializers};


And that will compile just with the current version of avr-gcc.   However, it generates an error on versions 4.6.2 and 4.7.2 that say something to the effect of "internal error:  report a bug ..."

Doing this

Code: [Select]
typedef struct {
        const char a[2];
        const char aardvark[9];
        //... etc 10k words
} words;

const words dictionary PROGMEM_blahblah = {
            .a="a",
            .aardvark="aardvark",
            //etc, 10000 initializers};


Will not compile on the current Arduino IDE but will compile on version 4.6.2 and v4.7.2.

I inevitably settled on a different means - and this isn't entirely debugged.  I'm still having trouble with the flash memory sections, but I did the following:

Code: [Select]

const char apple[6] PROGMEM_yadda="apple";
const char bear[5] PROGMEM_yadda="bear";
// etc. etc. etc. 10000 words

const char* dictionary[10000] PROGMEM_yadda+1 ={apple, bear, //etc etc



The idea is that 64k of dictionary data is in a chunk of flash designated by the attribute "PROGMEM_yadda" and an array of pointers to that data is in a chunk of memory called "PROGMEM_yadda+1"  (it may be obvious to most - but please don't try to create code with PROGMEM_yadda... that's just an example )  That compiles and links peachily.  

Now - PROGMEM what I learned about PROGMEM is that as an attribute it specifies putting data into flash.  Using PROGMEM is a multi-step process.  You have to put it into your code, explicitly.  But you also have to change the linker script to understand what you mean by that attribute.  And then retrieving data stored via PROGMEM requires using accessors of the form

Code: [Select]
pgm_read_word_far(&myArray[i]);



...continued
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 04, 2013, 06:18 pm
...from above

But the important thing to note is that PROGMEM designates that the linker put your data into a space that is 64k in size.  If you try to specify data bigger than 64k it might actually compile, but your results are going to be irratic - at least mine were using 1.0.3.  Another important thing to note is that the PROGMEM attribute puts the data in a certain place, and only in that place.  If you have more than 64k of data, you can put some where PROGMEM says and that spot is 64k big, but then where do you put the rest, and how do you get to it?

Now, just for the sake of discussion - there are lots of flash memory sections, used for different things.  You have the .text section, and the .data section, and lots of .fini1, .fini2,.finix...etc.  Code and data is marked by the __attribute__((section(".blah"))) tag, and then there is a corresponding note in the linker script that says what to do with things that are in section ".blah".  

Some have had success putting data in the section marked with the attribute ".fini7".   The AVR documentation says this is a user-definable section, and you can certainly use it to put a 64k chunk. If you have more than 64k you need more than one section.

Well, after much weeping and gnashing of teeth I got a lot of help from this:

http://www.avrfreaks.net/index.php?module=PNphpBB2&file=viewtopic&t=93874&highlight=

which explains how to set up several PROGMEM memory segments/sections. It pushes the static data into flash above the code itself, as is recommended by many.  It also provides some code to address those sections.  So, using that method which not only required using an include called  "morepgmspace.h" - and plz note this is a modified one from Carlos Lamas's original one, as well as making mods to the linker script found as  "avr6.x" - which is the script used for chips that have 256k of Flash.  Other chips will have different linker scripts.

With that combination of things,  I could specify constant global vars with the tags:  PROGMEM_SEG1, PROGMEM_SEG2, PROGMEM_SEG3.  I can them address them separately through a variety of means.  The linker script is modified to place the data in locations which begin at 0x10000, 0x20000, and 0x30000 respectively.  

Given that combo of linker script changes and includes I could do the following in code given the way I specified the data above:

Code: [Select]
 
char wordIGot[50];
uint_farptr_t theUltimateAddress  = GET_FAR_ADDRESS(myDictionary) + PROGMEM_SEGyadda_BASE_ADDRESS + (indexOfWord*sizeof(char*));

strcpy_Pyadda(wordIGot,pgm_read_byte_far(theUltimateAddress);



Note that I have not yet figured out which strcpy function to use with these addresses, so I wrote my own.  

apparently, this also works in some universe but I have not yet got it to work:

Code: [Select]

#include <morepgmspace.h>
char myWord[50];
int theIndexOfTheWordIWant;
strcpy_PF(myWord, pgm_get_word_far(&myDictionary[theIndexOfTheWordIWant]));


What I have been unable to accomplish at this point, is that I cannot retrieve the char* pointers from the array of 10000 pointers I specified as "dictionary[]" above.
Just for sake of example, I have stored 64k of strings in the memory noted with the attribute PROGMEM_SEG2.  I have stored the 10000 pointers to that data in PROGMEM_SEG1.

I am presuming these char* pointers are all 16-bit, so that they are addresses to the char strings all WITHIN  the same memory segment, and that I will have to do something of the order of:

Code: [Select]

#include <morepgmspace.h> // note this is the MODIFIED one available on avrfreaks.com, not the original from Carlos Lamas's website
uint16_t theLocalAddress = pgm_read_word_far(&dictionaryWhichIsInSeg1[which word do I want]);

//but as I mentioned the above line doesn't seem to work always, as you can't retrieve the dictionary address through variable indexing.
//In that case, you have to do this..

uint16_t theLocalAddress = pgm_read_word_far(GET_FAR_ADDRESS(dictionary) + PROGMEM_SEG1_BASE + whichWordDoIWant*sizeof(char*));

//but I'm not sure how well that works either



then

Code: [Select]
char* theRealAddress = pgm_read_word_far(theLocalAddress + PROGMEM_SEG2_BASE);


However, I  think I may be in some sort of compiler optimization hell.  It seems impossible to retrieve the pointers to the const char*s with pgm_read_word_far...but I can get to the words themselves, which are stored sequentially in PROGMEM_SEG2 and thus seem like one big long string.

I also ran into the perennial issue with the Mega2560 bootloader hanging...the new version of the bootloader hex does seem to work but I wind up with a problem on verification.  It gives me a warning, but the code loads just fine.

If anyone followed all that, thanks much for your valuable time.

With kind regards
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: nickgammon on Feb 04, 2013, 08:47 pm
I must admit I skimmed a bit, but I think I can explain something.

The raw assembler commands can, quite efficiently, index into 64 bytes of memory by using X, Y and Z registers which are 16-bit. Bear in mind it is basically an 8-bit processor. :)

Thanks to your post I spotted on page 2 of the assembler manual RAMPX, RAMPY and RAMPZ which can increase the addressing range. I presume though, that you have to decide in advance whether you want to access below 64K or above 64K. Reading further, though, that applies to SRAM not PROGMEM.

Further reading reveals ELPM, a variant on LPM (load from program memory) which appears to address 24-bits into program memory.

Presumably, to retrieve data from a large array, in program memory, the ELPM instruction has to be generated at some point.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 05, 2013, 12:04 am
Hi Nick
Yes, I have seen in the code <pgmspace.h> and also <morepgmspace.h> where for machines that support ELPM, the macros pgm_read_xxx_far are mapped into ELPM calls instead of LPM.  
However, I've yet to determine exactly how to use those.   It does seem to work just fine when I have a single 64k block of flash data tagged with the attribute PROGMEM.   I think this is true because the following works:

Code: [Select]

#include <pgmspace.h>
//...(other includes)
char myString[10];
const char abc[] PROGMEM = "abc";
const char def[] PROGMEM = "def";
const char* abcdef[] PROGMEM = {abc,def};
// ... stuff
for(int i=0;i<2;i++){
strcpy_P(myString,pgm_read_word(&abcdef[i]));
Serial.println(myString);
}

Note the use of strcpy_P is defined in <pgmspace.h>.   Note also that I can address the array with a var, which does not seem to be true if you try to create a var in memory space mapped through other means - like the terms PROGMEM_FAR and PROGMEM_SEGx as specified in <morepgmspace.h>   In those cases,  the compiler seems to want a static situation where you do something like;
Code: [Select]
 
long_address = GET_FAR_ADDRESS(abcdef) + BASE_ADDRESS_OF_MEMORY_SECTION + (which element)*sizeof(int);
strcpy_whatever(myString, pgm_read_word(long_address);


I suppose 24-bit addressing must exist (at least, 18-bit addressing must) exist even to accept an address in that upper 256k...

If I use multiple 64k flash sections or I just try to rely on 24-bit addressing by specifying
Code: [Select]

#include <morepgmspace.h>
const variable myVar[70k's worth]... a whole lot of stuff more than 64k

then this does not work...
Code: [Select]

int abc = pgm_read_word(&abcdef[i]);


So I am unsure of the occasions when 24-addressing is supported by the current compiler/linker, and when it's not...  As I said, it seems that with versions avr-gcc 4.6.2 and 4.7.2 it does not balk at the code which calls more than 64k.  But I haven't yet figured out if I can actually make it work with my app.  I still have to go back and de-Arduino-ize my Arduino code into pure AVR code to try that in another environment, like AVR-Eclipse or AVR-XCode, which support those compiler versions...

Cheers
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: pYro_65 on Feb 05, 2013, 06:11 am
Quote
However, I  think I may be in some sort of compiler optimization hell.  It seems impossible to retrieve the pointers to the const char*s with pgm_read_word_far


Take heed to the '64K boundary' talk, it is not an optimisation/bug but part of the design.

Pgm pointers are 16-bit word pointers.  As in, 64K unique addresses, 128K bytes addressed.
The 64Kb boundary for variable data is due to standard pointers addressing a single byte.

If you treat your upper memory as word aligned you could then change your addressing scheme, but then the flash gets swallowed twice as quick.



Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: westfw on Feb 05, 2013, 06:22 am
The way I think it fails is...
The linker puts things together like this:
  vectors
  progmem
  trampolines
  code

I think that this means that you'll have problems whenever the progmem data exceeds 64k (+/- a little bit), because the trampolines have to be in the first 64k (and I think the vectors can only access the first 64 or 128k)

I'm not sure why this is done; having progmem in the first 64k is "convenient", but having the trampolines and the startup code in the first 64k is NECESSARY.

The new 4.6.2 compiler seems to do the same thing.
I think 4.7 includes a 24bit pointer type, and a complete rework of how progmem is done, but it's still likely to require some special treatment to get large sketches to run.

In theory, the behavior can be changed with a custom linker script.

Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: westfw on Feb 05, 2013, 06:35 am
Quote
Pgm pointers are 16-bit word pointers.

Well, not quite.  The flash memory is organized as words, and program instructions are always aligned on a 16bit boundry.  The "jmp" and "call" instructions do not include a bit that differentiates which byte, so in theory a "pointer to function" could be a word pointer that addresses 128k of memory.  The bootloader and programming protocols use a lot of word pointers, which is why "burn bootloader" starts to require different tools beyond the 128k (byte) barrier rather than the 64k barrier.

However, the indirect jump (jump to the address contained in a register) AND the "load program memory" instruction (for reading data from flash) both take a full 16bit byte pointer; the low bit is ignored for the ijmp, and does bytewise addressing for LPM.  So most actual pointers really are byte pointers.
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: westfw on Feb 05, 2013, 06:42 am
Quote
I think the vectors can only access the first 64 or 128k

This part turns out to be wrong.  The two-word "jump" and "call" instructions can access the full 4MB address space.
However, the indirect jump/call (ijmp/icall) instructions can only access the first 64kbytes, even on CPUs with 128k of memory.  Chips with 256k of memory add an "EIJMP/EICALL" that can jump anywhere.

(This sort of "kludge" is why I think 32bit CPUs (like the ARM in Due) will displace 8bit CPUs in applications where code or data exceed 64k.)
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 05, 2013, 07:52 am
Hi and thanks all,
I did get my code to work tonite. I divided the data and pointers into 3 segments each of which is less than 64k.  There are 2 PROGMEM segments containing a bunch of explicitly initialized strings of the form:
Code: [Select]

#include <morepgmspace.h> // and associated mods to ldscript to accommodate the PROGMEM attrs
const char abcde[9] PROGMEM_SEG2 = "aardvark";


And also

Code: [Select]
const char fghij[4] PROGMEM_SEG1 = "ant";

And another segment full of pointers to those strings like this

Code: [Select]
const char* dictionary[10000] PROGMEM_SEG3= {abcde,fghij,..};

I can pull the chars out of the various segments like this

Code: [Select]
char* myWordInSeg2= GET_FAR_ADDRESS(dictionary[0]) + SEG2_OFFSET + index*sizeof(char*);
      strcpy_Pseg2(localCharArray,myWordInSeg2); // I wrote the copy to handle addressing from SEG a


It works but I'm durned if I can figure out why...somewhere some 18/24 bit addressing is taking place via ELPM... I may just be lucky because my strings are all loaded sequentially. The char ptrs in SEG3 seem to me to be useless. I think I'm just getting into seg2 and yanking out strings wantonly without guidance from the ptrs in the dictionary  array. But somehow this is working. The fact I don't understand it makes me nervous it is not a rigorous solution that only works temporarily.

Anyway.. Onward to Arduweenie land,
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: iceowl on Feb 06, 2013, 04:29 am
OK, the following works:

Using the multi-progmem-segment method

1) Store 64k of strings in PROGMEM_SEG2  (doesn't have to be exact or padded) denoted with a memory start location of 0x20000
2) Store ~64K of strings in PROGMEM_SEG3  denoted with memory start location of 0x30000
3) Store 20k of pointers to those strings in the lowest progmem space with PROGMEM_FAR which goes...I dunno where.

Plz note that you cannot deposit into memory a chunk of data larger than 64k, at least not with this version of avr-gcc/ld in the arduino 1.0.3.  I believe this will be different once we get to avr-gcc V4.7...someday...maybe.


The strings can be accessed thus:

Code: [Select]

#include <morepgmspace.h>

if(accessing a char* in a string at a location < size of the 64k of stuff in seg 2){

     unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[0]) +wordNumber*2); //wordNumber is the index into the array of strings called dictionary0
     strcpy_PX(newWord,index,PROGMEM_SEG2_BASE);
 
   }
   else {
     unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[0])+wordNumber*2); //note that the array dictionary0 also has data in SEG3!
     strcpy_PX(newWord,index,PROGMEM_SEG3_BASE);

   }

char* strcpy_PX(char* des, uint_farptr_t src,unsigned long base)
{
 unsigned long p = base + src;

 char* s = des;

 do {
   *s = pgm_read_byte_far(p++);
 }
 while(*s++);
 return des;
}




Note that you cannot access the data in those SEG2/SEG3 locations through specification of an indexed addressing scheme like

Code: [Select]
unsigned int index = pgm_read_word_far(GET_FAR_ADDRESS(dictionary0[i]);

I have been utterly unsuccessful in storing and reading out of the SEG 1 location which is denoted with a memory start location of 0x10000.  However, I can put 20k of char*s at wherever the attr PROGMEM_FAR puts it.  I have to go back to look at the linker script (avr6.x) to see where that's being mapped to.  I know it's going into .data (or maybe just past it...)

At least I understand what's happening now.   If I try to put anything into PROGMEM_SEG1 I not only get errors trying to read it back, but the bootloader gives me a verification error - which is probably the root cause of the whole problem.  I don't have it in me to look into the bootloader at this point...  On to another problem!

Cheers to all
Joe
Title: Re: Just a newby asking the 64k question again - Arduino Mega2560
Post by: StanleyAudio on Apr 03, 2013, 11:27 am
Hi,

Just joined in and digged out a lot of useful information, thanks

I am working on a project using Mega 2560 for voice announcement, so I need to store a large amount of compressed voices in the program memory. 256kB of flash on Mega2560 can give me about 7 minutes of voice playback.
So I encountered the same problem you guys have - accessing data stored outside the 64kb boundary of program memory.
Thanks pYro_65 for the neat solution of declaring PROGMEN_FAR in the program memory outside first 64kB, that solves the download and crash problem. Using the pgm_read_byte_far() I can read data anywhere in the 256kb program memory.
The GET_FAR_ADDRESS macro for converting program memory address to uint_farptr_t is wonderful, and with all these, I can get my program work; all 7 minutes of voices can be played without problem, thanks.

However, in tidying up my program and orgainzing my voice data in an easy indexing manor, I encountered another minor problem!

I need to store the address of each voice segment into an index table so that I can load the address and read the data accordingly. Of course, the index table contains 32 bit long address. The problem I have is; the GET_FAR_ADDRESS() macro works only in run time when it is called. It can not work in compile time, so when I build up my index table, done in compile time, the compiler doesn't allow me to cast the voice segment name (far address) into uint_farptr_t type.  My question is; anyone of you know how to set up a 32 bit pointer table that points to the far program memory in compile time?
Of course, I can still get my program works by writing a simple initialization rountine (run a sequence of GET_FAR_ADDRESS to iinitialize the table)
and get it run before my main code, but it will be a tidous job to type in all the voice segment names. I want the index table generated automatically in compile time. Any suggestion?

Thanks
Stan