How do I reference PROGMEM char arrays in my PROGMEM structs?

Hi,
I am making a bar bot. I pretty much have the hardware part down. I bought a touchscreen and I want to store mixed drink objects on my Arduino so I can display all the properties of a "drink" on the screen. I don't need help with the touch screen, this is a standard C++ question.

I found a big list of drinks online in JSON format. I want to represent these json objects as structs in my Arduino code as I believe that will be the easiest way to reference them and their properties. The JSON structure looks like this:

    {
        "idDrink": "17834",
        "strDrink": "Abbey Cocktail",
        "strCategory": "Ordinary Drink",
        "strIngredient1": "Gin",
        "strIngredient2": "Orange bitters",
        "strIngredient3": "Orange",
        "strIngredient4": "Cherry",
        "strMeasure1": "1 1/2 oz ",
        "strMeasure2": "1 dash ",
        "strMeasure3": "Juice of 1/4 ",
        "strMeasure4": "1 "
    }

There are a LOT of drinks in this list. So to save on memory I am trying to store them in PROGMEM, because even a mega can't handle the # of drinks I want to store. One memory saving technique I want to try is to first store a list of unique ingredient strings as a lot of the drinks share the same ingredients and I figured a reference to a char array would take up less memory than the string "vodka" duplicated 40+ times. I had no problem making the ingredients array:

const char string_54[] PROGMEM = "Blue Curacao";
const char string_55[] PROGMEM = "Blueberry Schnapps";
const char string_56[] PROGMEM = "Bourbon";
const char string_57[] PROGMEM = "Brandy";
...
const char *const ingredients[] PROGMEM = {
    string_54,
    string_55,
    string_56
    string_57
    ...
};

But I am having trouble referencing these strings in my "drink" structs (Which I also want to store in PROGMEM to save on memory). So far the only way I have gotten PROGMEM structs working is like this:

// I know this doesn't match the JSON structure from above, I am just trying to get an MVP working here first
struct MixedDrink {
  int id;
  char* ing1;
};
// Then instance a whole bunch of "MixedDrink"s in an array:
const MixedDrink drinksPROGMEM[] PROGMEM =
{
  {1, "foo"},
  {2, "String2"},
};
// MixedDrink struct in SRAM we will constantly overwrite with one from PROGMEM
MixedDrink drinksSRAM;
// Then here is an example of how I have been accessing them in the loop.
void loop() {
  delay(500);
  for( int i=0; i<(sizeof(drinksPROGMEM)/sizeof(MixedDrink)); i++)
  {
    memcpy_P( &drinksSRAM, &drinksPROGMEM[i], sizeof(MixedDrink));
    Serial.println( drinksSRAM.id);  // display prob A
    Serial.println( drinksSRAM.ing1);  // display prob B etc.
  }
}

Could someone please show me how I would reference one of the ingredients[] strings in my drinksPROGMEM[] definitions? I know this doesn't work but what I am trying to achieve is something like this:

const char string_999[] PROGMEM = "My Ingredient String";

const char *const ingredients[] PROGMEM = {
    string_999
}

struct MixedDrink {
  int id;
  char* ing1;
};

const MixedDrink drinksPROGMEM[] PROGMEM =
{
  {1, ingredients[0]},
};
MixedDrink drinksSRAM;
void loop() {
  delay(500);
  // would only loop once in this example but whatever
  for( int i=0; i<(sizeof(drinksPROGMEM)/sizeof(MixedDrink)); i++)
  {
    memcpy_P( &drinksSRAM, &drinksPROGMEM[i], sizeof(MixedDrink));
    Serial.println( drinksSRAM.id);  // displays 1
    Serial.println( drinksSRAM.ing1);  // displays "My Ingredient String"
  }
}

When you have a .print() statement with a char* that is referencing PROGMEM, you need to cast the char* to __FlashStringHelper* to let the compiler know the reference is to PROGMEM instead of ram.

const char string_999[] PROGMEM = "My Ingredient String";

const char *const ingredients[] PROGMEM = {
    string_999
};

struct MixedDrink {
  int id;
  const char* ing1;
};

const MixedDrink drinksPROGMEM[] PROGMEM =
{
  {1, ingredients[0]},
};
MixedDrink drinksSRAM;

void setup() {
  Serial.begin(115200);
}

void loop() {
  delay(500);
  // would only loop once in this example but whatever
  for( int i=0; i<(sizeof(drinksPROGMEM)/sizeof(MixedDrink)); i++)
  {
    memcpy_P( &drinksSRAM, &drinksPROGMEM[i], sizeof(MixedDrink));
    Serial.println( drinksSRAM.id);  // displays 1
    Serial.println((__FlashStringHelper*) drinksSRAM.ing1);  // displays "My Ingredient String"
  }
}

@david_2018 while your program compiles it does not work. The line:
Serial.println((__FlashStringHelper*) drinksSRAM.ing1);  // displays "My Ingredient String" displays gibberish in the console, not "My Ingredient String". I don't think the chars are being loaded correctly or it could be that the original reference:

{1, ingredients[0]}

Is not valid. ingredients itself is PROGMEM so maybe a simple ingredients[0] reference won't do. Thanks for the interest though.

Which arduino board are you using, and which version of the Arduino IDE?

Also note that my code had the Serial baud rate set to 115200, that needs to match the baud rate set in the serial monitor or you will see gibberish.

rorosentrater:
@david_2018 while your program compiles it does not work.

Works for me on an Uno with IDE v1.8.12.

However, I would change the 'for' loop index to an unsigned type to fix the compiler warning:

for ( uint16_t i = 0; i < (sizeof(drinksPROGMEM) / sizeof(MixedDrink)); i++)

Oh my bad guys sorry. Yeah it totally works. I had the monitor on the wrong baudrate. Thanks this is exactly what I needed.

I don't think the chars are being loaded correctly or it could be that the original reference:

{1, ingredients[0]}

Is not valid. ingredients itself is PROGMEM so maybe a simple ingredients[0] reference won't do. Thanks for the interest though.

You will likely find that the sample code will not work once you increase the size of the arrays. The compiler is quite aggressive when optimizing code, and will substitute a direct reference to the string instead of going through the single element array.

@david_2018 I did not have problems with the compile per se, but I did run into an issue where I am going over my Global variables memory limit once I increased the array sizes. This is what I have:

// There are ~350 of these
const char string_0[] PROGMEM = "151 Proof Rum";
const char string_1[] PROGMEM = "7-Up";
...
// Likewise there are ~350 of these
const char *const ingredients[] PROGMEM = {
    string_0,
    string_1,
    ...
};

// I know the struct properties look "out of order" here but I am writing this C++ code...
// With a python script as there is just so much text. There's no way I could do this by hand.
// The python script is able to keep the order of these props straight in the instances below don't worry.
struct MixedDrink {
    int id;
    const char* strDrink;
    const char* strTags;
    const char* strCategory;
    const char* strIngredient1;
    const char* strIngredient2;
    const char* strIngredient3;
    const char* strIngredient4;
    const char* strIngredient5;
    const char* strMeasure1;
    const char* strMeasure2;
    const char* strMeasure3;
    const char* strMeasure4;
    const char* strIngredient6;
    const char* strMeasure5;
    const char* strMeasure6;
    const char* strIngredient7;
    const char* strIngredient8;
    const char* strIngredient9;
    const char* strMeasure7;
    const char* strMeasure8;
    const char* strIngredient10;
    const char* strIngredient11;
    const char* strMeasure9;
    const char* strMeasure10;
    const char* strMeasure11;
    const char* strIngredient12;
    const char* strMeasure12;
};

// There are roughly ~500 of these. This is what I think is eating the most memory
// The MixedDrink objects are long so ill only give 1 example
const MixedDrink drinksPROGMEM[] PROGMEM =
{
    {11000, "Mojito", "IBA,ContemporaryClassic,Alcoholic,USA", "Cocktail", ingredients[214], ingredients[217], ingredients[318], ingredients[240], ingredients[308], "2-3 oz ", "Juice of 1 ", "2 tsp ", "2-4 ", "", "", "", "", "", "", "", "", "", "", "", "", "", "", ""}
};

MixedDrink drinksSRAM;

void setup() {
  Serial.begin(9600);
}

void loop() {
  delay(500);
  for( int i=0; i<(sizeof(drinksPROGMEM)/sizeof(MixedDrink)); i++)
  {
    delay(500);
    memcpy_P( &drinksSRAM, &drinksPROGMEM[i], sizeof(MixedDrink));
    Serial.println("--------------------");
    Serial.println( drinksSRAM.id);
    Serial.println(drinksSRAM.strDrink);
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient1);  // displays "My Ingredient String"
  }
}

If I cut down the number of defined MixedDrink objects, I can run the code and the loop reliably prints the id, drink name (strDrink) and first ingredient. So, it works, I just need to bring down the memory usage. Do you see an easy saving?

I think the strings that are defined in the MixedDrink instances with actual "quotes" are being stored in the global variable memory, which is what's taking up so much space. These remaining strings are fairly unique though. strDrink (the drink name) especially. I don't know if storing those in a PROGMEM array will reduce the global variable memory usage.

You are correct, the quoted text in the MixedDrink array is being stored in ram, not PROGMEM. You either have to define separate char arrays for each string, or reserve enough space in the struct to store the actual char array there (not a good idea unless all the text is the same length, otherwise you waste a lot of memory) - that will also run you into the hard limit of 32,767 bytes in an array.

I would not create an array for ingredients, naming the char arrays for the ingredients with the actual ingredient makes it much clearer what you are referencing, and also makes it much easier to use a text editor to generate the C++ code from the JSON list that you have, so that you don't have to physically type everything in.

Here is a good explanation on putting constant data into program memory.

I modified your latest code to give an idea of how I might do it:

// There are ~350 of these
const char ingredient_151_Proof_Rum[] PROGMEM = "151 Proof Rum";
const char ingredient_7_UP[] PROGMEM = "7-Up";
const char ingredient_Sugar[] PROGMEM = "Sugar";
const char ingredient_Tobasco[] PROGMEM = "Tobasco";
const char ingredient_Everclear[] PROGMEM = "Everclear";
const char ingredient_Salt[] PROGMEM = "Salt";

const char drink_Mojito[] PROGMEM = "Mojito";
const char tag_IBM_ContemporaryClassic_Alcoholic_USA[] PROGMEM = "IBA,ContemporaryClassic,Alcoholic,USA";
const char category_Cocktail[] PROGMEM = "Cocktail";

const char measure_2_3_oz[] PROGMEM = "2-3 oz ";
const char measure_Juice_of_1[] PROGMEM = "Juice of 1 ";
const char measure_2tsp[] PROGMEM = "2 tsp ";
const char measure_2_4[] PROGMEM = "2-4 ";

struct MixedDrink {
  int id;
  const char* strDrink;
  const char* strTags;
  const char* strCategory;
  const char* strIngredient1;
  const char* strIngredient2;
  const char* strIngredient3;
  const char* strIngredient4;
  const char* strIngredient5;
  const char* strMeasure1;
  const char* strMeasure2;
  const char* strMeasure3;
  const char* strMeasure4;
  const char* strIngredient6;
  const char* strMeasure5;
  const char* strMeasure6;
  const char* strIngredient7;
  const char* strIngredient8;
  const char* strIngredient9;
  const char* strMeasure7;
  const char* strMeasure8;
  const char* strIngredient10;
  const char* strIngredient11;
  const char* strMeasure9;
  const char* strMeasure10;
  const char* strMeasure11;
  const char* strIngredient12;
  const char* strMeasure12;
};

const char blank[] PROGMEM = ""; //empty line to fill the unused char* entries of the struct

// There are roughly ~500 of these. This is what I think is eating the most memory
// The MixedDrink objects are long so ill only give 1 example
const MixedDrink drinksPROGMEM[500] PROGMEM = {
  { 11000,
    drink_Mojito,
    tag_IBM_ContemporaryClassic_Alcoholic_USA,
    category_Cocktail,
    ingredient_7_UP,
    ingredient_Salt,
    ingredient_Tobasco,
    ingredient_Everclear,
    ingredient_Sugar,
    measure_2_3_oz,
    measure_Juice_of_1,
    measure_2tsp,
    measure_2_4,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank,
    blank
  }
};

MixedDrink drinksSRAM;

void setup() {
  Serial.begin(9600);
}

void loop() {
  delay(500);
  for ( unsigned int i = 0; i < (sizeof(drinksPROGMEM) / sizeof(MixedDrink)); i++)
  {
    delay(500);
    memcpy_P( &drinksSRAM, &drinksPROGMEM[i], sizeof(MixedDrink));
    Serial.println("--------------------");
    Serial.println( drinksSRAM.id);
    Serial.println((__FlashStringHelper*) drinksSRAM.strDrink);
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient1);  // displays "My Ingredient String"
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient2);
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient3);
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient4);
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient11); //demonstrates what prints out from the 'blank' entry
    Serial.println((__FlashStringHelper*) drinksSRAM.strIngredient5);
    delay(600000); //used to keep output from scrolling random output from uninitialized array elements
  }
}

If you have a TFT screen, doesn't it have a SD drive? I'd think that would be a LOT easier to deal with. Then you could store as many as you wanted on the Drive and just maintain in RAM what you need for your scrolling selection list. Etc.

Like a contact's list for a smartPhone.

Or not..

-jim lee

Storing the data on the SD card would be a lot simpler to implement, particularly if there is more text than will fit within the first 64K of PROGMEM. Accessing PROGMEM on a Mega past the 64K boundary becomes more complicated.

To the OP, is the JSON list of drinks that you are using downloaded from a website? I'd like to see what the actual data looks like.

david_2018:
Storing the data on the SD card would be a lot simpler to implement, particularly if there is more text than will fit within the first 64K of PROGMEM.

While you're at it, why not upgrade to a processor board with more resources? The Teensy 3.6 has a built-in Micro SD card slot, lots of RAM / Flash, and a fast 32-bit ARM processor.