PROGMEM and accessing an array of structs

Using Arduino Programming Questions
1

Hi, I'm having some trouble using PROGMEM with a big static array. From the tutorials I know that any reads need to be done with the pgm_read functions, but how do those work when the array consists of structs instead of some raw datatype such as "int" or "byte"?

I'm sending a bunch of information about the current program state over a data radio one by one due to performance reasons. This is basically just a iterable list of pointers and some information about them. It works quite fine, but having it in memory is wasteful.

The current array is this:

const static remoteSetting toSend[] = {                                    //This is the struct that has pointers to the plane's settings
  //   modifiable    data type         pointer to data
  {O |  (1 << mod) | (1 << isfloat), data : &elevator._kp},
  {O |  (1 << mod) | (1 << isfloat), data : &elevator._ki},
  {O |  (1 << mod) | (1 << isfloat), data : &elevator._kd},
  {O |  (0 << mod) | (1 << isfloat), data : &elevator._error},
  {O |  (0 << mod) | (1 << isfloat), data : &elevator._ITerm},
  {O |  (1 << mod) | (1 << isfloat), data : &rudder._kp},
  {O |  (1 << mod) | (1 << isfloat), data : &rudder._ki},
  {O |  (1 << mod) | (1 << isfloat), data : &rudder._kd},
  {O |  (0 << mod) | (1 << isfloat), data : &rudder._error},
  {O |  (0 << mod) | (1 << isfloat), data : &rudder._ITerm},
  {O |  (1 << mod) | (1 << isint), data : targetPitot},
  {O |  (1 << mod) | (1 << isint), data : rollSlackAngle},
  {O |  (1 << mod) | (1 << isint), data : rollResponse},
  {O |  (1 << mod) | (1 << isint), data : pitchResponse},
  {O |  (1 << mod) | (1 << isint), data : maxPitchAngle},
  {O |  (1 << mod) | (1 << isint), data : maxYawRate},
  {O |  (1 << mod) | (1 << isint), data : &rthAltitude},
  {O |  (1 << mod) | (1 << isint), data : &waypointAcceptRadius},
  {O |  (1 << mod) | (1 << isint), data : &lowBatt},
  {O |  (1 << mod) | (1 << isint), data : &critBatt},
  {O |  (0 << mod) | (1 << isbyte), data : &controlMode},
  {O |  (1 << mod) | (1 << isbyte), data : &motorOverride},
  {O |  (0 << mod) | (1 << isint), data : &orientation.velocity},
  {O |  (0 << mod) | (1 << isint), data : &orientation.yawrate},
  {O |  (0 << mod) | (1 << isint), data : &orientation.rollrate},
  {O |  (0 << mod) | (1 << isbyte), data : &system},
  {O |  (0 << mod) | (1 << isbyte), data : &gyro},
  {O |  (0 << mod) | (1 << isbyte), data : &accel},
  {O |  (0 << mod) | (1 << isbyte), data : &mag},
  {O |  (0 << mod) | (1 << isbyte), data : &motorArmed},
  {O |  (0 << mod) | (1 << isbyte), data : &fixStatus},
  {O |  (0 << mod) | (1 << isfloat), data : &lat},
  {O |  (0 << mod) | (1 << isfloat), data : &lon},
  {O |  (0 << mod) | (1 << isbyte), data : &debugByte},
  {O |  (0 << mod) | (1 << isint), data : &debugInt},
  {O |  (0 << mod) | (1 << isfloat), data : &debugFloat},
  {O |  (1 << mod) | (1 << isint), data : &targetpoint->altitude},
  {O |  (0 << mod) | (1 << isuint), data : &loopsPerTransmit},
  {O |  (0 << mod) | (1 << isuint), data : &loopsPerSecond},
  {O |  (0 << mod) | (1 << isuint), data : &distToCurrent}
};

The struct used is as follows:

struct remoteSetting{      //Determines the values that are communicated between plane and ground station
  byte type;      // is float | is ulong | is long | is uint | is int | is char | is byte | is modifiable
  void *data;      //Void pointer to whatever, type is interpreted by the type identifier byte.
  };

I'm at a loss what would be the best way to type this out neatly and access the array from PROGMEM. As these are all static while the program runs, I'd want the structs themselves, not only the array, to reside in flash. The only way I know of involves making the (progmem) structs first, then making another (progmem) array of pointers to point there, which seems a bit superfluous.

2

You use the address of the structure variable as the address to the pgm functions.

pgm_read_byte( &toSend[2].type );


//Pointers are two bytes, so use the 'word' version
pgm_read_word( &toSend[5].data );
3

Thanks, but just changing the list initialization to "const static remoteSetting toSend[] PROGMEM =" causes the arduino to reboot endlessly even when the list is never referenced so I can't yet test that. I'd think there should be no effect since the list is never referenced.

4

Jopj:
Thanks, but just changing the list initialization to "const static remoteSetting toSend[] PROGMEM =" causes the arduino to reboot endlessly even when the list is never referenced so I can't yet test that. I'd think there should be no effect since the list is never referenced.

Try it in a separate test sketch. I'd assume (as I can't see your code) that there is something else causing your problem. Most probably when you access the pointers you read from flash. Are they in PROGMEM also?

This is a quick example showing you it works.

struct Foo{
  byte a;
  int b;
};

const Foo bar[] PROGMEM = {
  { 1, b: 1000 },
  { 2, b: 2000 },
  { 3, b: 3000 },
};

void setup() {
  Serial.begin(9600);

  for( const Foo &foo : bar ){
    Serial.print( pgm_read_byte( &foo.a ) );
    Serial.print( ", " );
    Serial.println( pgm_read_word( &foo.b ) );
  }
}

void loop() {}

The specific initialization of the second member is kind of unnecessary as you are using all members. This kind of initialization is useful when you do not want to initialize parameters in between others.

5

I got the example to work, and my own list works as well.. provided it never has more than 5 elements. Unless I'm mistaken, there is no specific size limit to what you can use progmem with. The 5 elements can be any on the list so those seem to work. I'm not accessing anything in the list as I'm just trying to put it in progmem without crashing everything first :slight_smile:

There is plenty of flash and ram space left, ~50% both, and pretty much everything is commented out anyway to debug this. I'm using arduino 1.6.4 if that has any bearing.

I included the (misleadingly named) code itself, but it's fairly long and I don't expect anyone to actually trawl through all that..

displaytest.ino (71.2 KB)

6

Its very late where I'm at (damn early... 5am), so I can look at it a bit later. However, the max length for any object/array in RAM or PROGMEM is 32k bytes (set by avr-gcc). I'm sure we can get this knuckled out.

7

Thank you for your responses!

I'm getting extremely odd behaviour, when increasing the size of the PROGMEM array. Code from "setup()" gets run repeatedly, but the board does not reboot since any debug counter in the loop keeps growing as normal. Even with a stripped down test sketch that never does anything with any pins, connected leds and beepers go crazy. I have only seen this kind of behaviour when running out of ram, but checking free memory inside the only function in loop() yields almost 1k free.

I too ran into some comments about a max size of the progmem arrays, but surely 7 instances of a struct with 5 bytes in it (35 bytes + overhead?) should not hit that limit? Especially since this very array works well without the progmem.

I am stumped.

8

I made a small test program that illustrates this problem. On my 328p Nano this simple sketch runs into problems (it spams the test print very fast despite the delay(1000), and pin 17 mostly stays high, occasionally going low. According to the IDE (1.6.4) it uses only 8% and 9% of flash and RAM respectively.

This is looking to be trickier than I thought.

#define mod 0      //These are bit values for the data types
#define isbyte 1
#define isloadable 2
#define isint 3
#define isuint 4
#define islong 5
#define isulong 6
#define isfloat 7


struct remoteSetting{      //Determines the values that are communicated between plane and ground station
  byte type;      // is float | is ulong | is long | is uint | is int | is char | is byte | is modifiable
  void *data;      //Void pointer to whatever, type is interpreted by the type identifier byte.
  };
  
  
byte O = 0;
float a = 444;

const static remoteSetting toSend[] PROGMEM ={                                    //This is the struct that has pointers to the plane's settings
  //  modifiable    data type         pointer to data
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  {O |  (1 << mod) | (1 << isfloat), data : &a},
  };

void setup() {
  // put your setup code here, to run once:
  Serial.begin(19200);
  pinMode(17, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
  Serial.println("asdasd");
  digitalWrite(17,!digitalRead(17));
  delay(1000);
}

edit: it's curious how it happens even when the compiler presumambly optimizes the whole array away since it's never used and increasing it's size doesn't affect program size.

another edit: the culprit might just be those bitshifts, I cannot replicate the problem with them replaced by a typed constant. I'd think since everything is known at compile time, they'd work like a constant in the first place, yet it doesn't seem so. If this is the case, this array will become very verbose, especially since it seems that to get the whole thing to PROGMEM I need to first put the structs there, and then make the PROGMEM array with pointers to them.

This seems silly and has additional overhead, there must be a better way :slight_smile:

9

If you want the variable 'O' to be a constant you should probably declare it 'const'

10

Good catch, that actually does seem to fix that small example program. I'll admit I don't fully understand how it being a normal variable would cause this as nothing actually writes to it.

It appears that just using PROGMEM on the array itself doesn't put the whole thing there, since some ram is still consumed (only array, not the structs inside that?). In the reference about PROGMEM strings were first declared indipendently into PROGMEM, then pointers to those in a PROGMEM array. In my case that would be clunky, is there a way to do that all at once?

11

Jopj:
It appears that just using PROGMEM on the array itself doesn't put the whole thing there, since some ram is still consumed (only array, not the structs inside that?).

Did you try the sketch I posted above?

The array is in PROGMEM, not RAM. Two bytes of RAM is used to store the pointer to PROGMEM.