Help with using PROGMEM with my defined structs

Projects Programming
1

My project has a bunch of static data (a school schedule for an entire school year), which uses arrays of structs for organization. It seems as though I can add the PROGMEM keyword to those definitions, and the results from the compiler certainly suggest that those data have been moved to PROGMEM. However, my understanding is that data that is in PROGMEM cannot simply be referenced the normal way. Rather, you must use primitive functions like pgm_read_word_near and pgm_read_byte_near (is there a form for a 32-bit quantity?). Well, there goes my well-designed data structures.

I am aware of the Flash library by Mikal Hart, but his doc says it supports strings, arrays, tables, and string arrays. Doesn't sound like it would support arbitrary structures.

My current thinking is to turn my simple C-style structures into classes, and implement "get" type methods for the data. Those methods could hide all the ugly address arithmetic that must be done to use the native data access functions, so that the bulk of the code would still look reasonable.

My questions are:

  1. Does this seem like a reasonable approach?
  2. Are there better alternatives?

Thanks!

2

It's not really that hard. Here is an example:

// magic8ballForUno010919.ino 3024/258 01/09/2019

// the official "Magic 8 Ball™" phrases
const char yes1[] PROGMEM =  "   WITHOUT A DOUBT";
const char yes2[] PROGMEM =  "   YOU MAY RELY ON IT";
const char yes3[] PROGMEM =  "   YES DEFINITELY";
const char yes4[] PROGMEM =  "   SIGNS POINT TO YES";
const char yes5[] PROGMEM =  "   IT IS DECIDEDLY SO";
const char yes6[] PROGMEM =  "   AS I SEE IT - YES";
const char yes7[] PROGMEM =  "   YES";
const char yes8[] PROGMEM =  "   MOST LIKELY";
const char yes9[] PROGMEM =  "   IT IS CERTAIN";
const char yes10[] PROGMEM = "   OUTLOOK GOOD";
const char no1[] PROGMEM =   "   MY SOURCES SAY NO";
const char no2[] PROGMEM =   "   DONT COUNT ON IT";
const char no3[] PROGMEM =   "   VERY DOUBTFUL";
const char no4[] PROGMEM =   "   OUTLOOK NOT SO GOOD";
const char no5[] PROGMEM =   "   MY REPLY IS NO";
const char idk0[] PROGMEM =  "   BETTER NOT TELL YOU NOW";
const char idk1[] PROGMEM =  "   ASK AGAIN LATER";
const char idk2[] PROGMEM =  "   REPLY HAZY TRY AGAIN";
const char idk3[] PROGMEM =  "   CANNOT PREDICT NOW";
const char idk4[] PROGMEM =  "   CONCENTRATE AND ASK AGAIN";

const char* const theAnswers[] PROGMEM = { yes1, yes2, yes3, yes4, yes5, yes6, yes7, yes8, yes9, yes10,
                                           no1, no2, no3, no4, no5, idk0, idk1, idk2, idk3, idk4,
                                         };

char phrase[64];

boolean inputState = false;
boolean lastInputState = false;

const byte inputPin = 2;  // pin -- button -- gnd pin
const byte gndPin = 4;  // my button is between pins 2 and 4
const byte randomPin = A0;

void setup() {
  Serial.begin(115200);
  pinMode(inputPin, INPUT_PULLUP);
  pinMode(gndPin, OUTPUT);
  digitalWrite(gndPin, LOW);
  randomSeed(analogRead(randomPin));
}

void loop() {

  // cheap button debouncer
  inputState = !digitalRead(inputPin);
  delay(1);

  // wait for the button to be pressed
  inputState = !digitalRead(inputPin);
  if (inputState != lastInputState) {
    lastInputState = inputState;

    // if the button is pressed, pick an answer
    if (inputState) {
      byte answer = random(0, 19);

      // build a string with PROGMEM function pgm_read_word() and print it
      strcpy_P(phrase, (char*)pgm_read_word(&(theAnswers[answer])));
      strcat(phrase, "\n");
      Serial.println(phrase);
    }
  }
}
3

Indeed, there's a whole library of functions to do all sorts of PROGMEM stuff. Check out the AVR pgmspace utilities.

4

Doug- thanks for the link.

Chris-

Most of the examples, like yours, are string examples, and I'm having a hard time relating that to my use case. I have a structure,

typedef struct _SingleDay
{
  uint16_t    Y;
  uint8_t     M;
  uint8_t     D;
  BellSched*  dayType;
  uint8_t     dayLetter;
} SingleDay;

and a global array of 180 of these called TheCalendar:

const PROGMEM SingleDay TheCalendar[]=
{
  {2018,9,4,NormalDay,'A'},
  {2018,9,5,NormalDay,'B'},
  {2018,9,6,NormalDay,'C'},
  {2018,9,7,NormalDay,'D'},
  ...
}

The BellSched structure is defined thus:

typedef struct _BellSched
{
  uint8_t NumPeriods;
  Period Periods[MAX_PERIODS];
} BellSched;

where Period is:

typedef struct _Period
{
  uint8_t  begH;
  uint8_t  begM;
  uint8_t  endH;
  uint8_t  endM;
  uint32_t aCol;
  uint32_t bCol;
  uint32_t cCol;
  uint32_t dCol;
} Period;

So that I can look through the table to find the current day, find the day's letter (A-D), and the day's bell schedule (e.g. NormalDay)

const PROGMEM BellSched NormalDay[]=
{
  9,
  { 8, 0, 8,50, OUTOFCLUSTER, RED,          OUTOFCLUSTER, YELLOW},
  { 8,52, 9,40, RED,          YELLOW,       YELLOW,       OUTOFCLUSTER},
  { 9,42, 9,52, ASPIRE,       ASPIRE,       ASPIRE,       ASPIRE},
  { 9,54,10,42, YELLOW,       OUTOFCLUSTER, RED,          RED},
  {10,44,11,32, GREEN,        OUTOFCLUSTER, BLUE,         GREEN},
  {11,34,12,23, OUTOFCLUSTER, BLUE,         GREEN,        PURPLE},
  {12,25,12,46, LUNCH,        LUNCH,        LUNCH,        LUNCH},
  {12,48,13,36, BLUE,         PURPLE,       PURPLE,       OUTOFCLUSTER},
  {13,38,14,26, PURPLE,       GREEN,        OUTOFCLUSTER, BLUE},
};

I have code that runs when the minute changes to see if we're in a new period:

void DoNewMinuteStuff()
{
  currentMinute = now.minute();
  if ( dayType == dtWeekend || dayType == dtHoliday || clockType == ctAfterSchool );
  else
  {
    //check against bellschedule, set pointer to current period, and set clock type
    BellSched *bs = Today->dayType;
    if ( clockType == ctBeforeSchool )
    {
      // see if we've made it to first period
      Period *p = &bs->Periods[0];
      if ( now.hour() == p->begH && now.minute() == p->begM )
      {
        currentPeriod = 0;
        clockType = ctDuringClass;
      }
    }
    else
    {
      Period *p = &bs->Periods[currentPeriod];
      if ( currentPeriod < bs->NumPeriods-1 )
      {
        Period *np = &bs->Periods[currentPeriod+1];
        // consider we may have started the next period
        if ( isAfterTime(now.hour(), now.minute(), np->begH, np->begM) )
        {
          clockType = ctDuringClass;
          currentPeriod++;
          return;
        }
      }
      if ( timeDiff(p->endH, p->endM, now.hour(), now.minute())  < countdownM )
      {
        clockType = ctEndFlash;
      }
      else if ( isAfterTime(now.hour(), now.minute(), p->endH, p->endM) )
      {
        if ( currentPeriod < bs->NumPeriods-1 )
        {
          clockType = ctPassing;
        }
        else clockType = ctAfterSchool;
      }
    }
  }
}

So for each field I want to retrieve, do I need to use the pgm_read_ function call? Seems so. But I don't want to have all those calls sprinkled throughout my code. I'd rather hide that complexity, so redefining things as a class, and using a "get" type method for accessing the data, seems the most natural. I believe, for instance, that strcpy_P hides the complexity of reading each byte in pgm space with a function call.

5

tastewar:
My current thinking is to turn my simple C-style structures into classes, and implement "get" type methods for the data. Those methods could hide all the ugly address arithmetic that must be done to use the native data access functions, so that the bulk of the code would still look reasonable.

My questions are:

  1. Does this seem like a reasonable approach?
  2. Are there better alternatives?

Thanks!

  1. Yes. Hiding the access to the flash will certainly make your program more readable and thus reduce the probability of introducing bugs.
  2. Wow, this is a huge amount of data (about 33kB?). Maybe you want to do a little bit of compression on these datasets?

tastewar:
So for each field I want to retrieve, do I need to use the pgm_read_ function call?

Since the arduino uses seperate data and flash address spaces there is no other way, than using pgm_read_xxx.

6

Take a look at memcpy_P(). But, because your struct contains pointers to other structs, you'll probably have to call it iteratively to affect a deep copy.

7

Thank you, LightuC.

Sketch uses 16380 bytes (50%) of program storage space. Maximum is 32256 bytes.
Global variables use 581 bytes (28%) of dynamic memory, leaving 1467 bytes for local variables. Maximum is 2048 bytes.

So I don't appear to be in need of compression at the moment, but curious about exactly what you're thinking.

The first thing I would do is remove the day letter from the table, as it is simply a regular 4 day cycle, and the letter can thus be inferred from the day number. I could store a 2 digit year and save a byte from the year field, but I lived through the Y2K crisis!. There are repeated patterns of colors, and I suppose I could add another level of indirection and only store each unique color sequence (from the Period struct) once, and keep pointers to those.

Happy to hear other ideas, though!

8

tastewar:
but I lived through the Y2K crisis!.

Do you think you'll be around to worry about the Y3K crisis 8)

9

tastewar:

Sketch uses 16380 bytes (50%) of program storage space. Maximum is 32256 bytes.

Global variables use 581 bytes (28%) of dynamic memory, leaving 1467 bytes for local variables. Maximum is 2048 bytes.




So I don't appear to be in need of compression at the moment, but curious about exactly what you're thinking.

Well, since you are only using 50% of your flash memory I wouldn't do anything at the moment. My rough calculation was based on the following:

sizeof(_SingleDay) = 7 Bytes
180 of these in the calender array = 1260 Bytes
for each day there is a BellSchedule with MAX_PERIODS (I assumed 9).
sizeof(BellSchedule) = 20 Bytes * 9 = 180
Again 180 of these = 32400

But you might only have a few schedules.

tastewar:
The first thing I would do is remove the day letter from the table, as it is simply a regular 4 day cycle, and the letter can thus be inferred from the day number. I could store a 2 digit year and save a byte from the year field, but I lived through the Y2K crisis!. There are repeated patterns of colors, and I suppose I could add another level of indirection and only store each unique color sequence (from the Period struct) once, and keep pointers to those.

Happy to hear other ideas, though!

I would probably only store a number for the color and one lookup table. This will save you 16 * MAX_PERIODS Bytes for each BellSchedule you have. However, since you are not even close to the limit, you might want to postpone that to later.

10

gfvalvo:
Do you think you'll be around to worry about the Y3K crisis 8)

Hoped it was obvious I was being silly there :slight_smile:

11

LightuC:
But you might only have a few schedules.

True! And the reason for the indirection. There is a normal day schedule, and 3 different early release schedules (seems excessive, but there it is)

Thanks again, LightuC!