Question about PROGMEM (strcpy_P usage)

Projects Programming
1

I'm in an "out of RAM" situation so I have use PROGMEM for const char arrays (basically these arrays are AT commands for a GSM shield).

My question is about the related PROGMEM tutorial in Arduino Documentation, specifically about this code section:

/*
      PROGMEM string demo
      How to store a table of strings in program memory (flash),
      and retrieve them.

      Information summarized from:
      http://www.nongnu.org/avr-libc/user-manual/pgmspace.html

      Setting up a table (array) of strings in program memory is slightly complicated, but
      here is a good template to follow.

      Setting up the strings is a two-step process. First, define the strings.
    */

    #include <avr/pgmspace.h>
    const char string_0[] PROGMEM = "String 0"; // "String 0" etc are strings to store - change to suit.
    const char string_1[] PROGMEM = "String 1";
    const char string_2[] PROGMEM = "String 2";
    const char string_3[] PROGMEM = "String 3";
    const char string_4[] PROGMEM = "String 4";
    const char string_5[] PROGMEM = "String 5";


    // Then set up a table to refer to your strings.

    const char *const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

    char buffer[30];  // make sure this is large enough for the largest string it must hold

    void setup() {
      Serial.begin(9600);
      while (!Serial);  // wait for serial port to connect. Needed for native USB
      Serial.println("OK");
    }


    void loop() {
      /* Using the string table in program memory requires the use of special functions to retrieve the data.
         The strcpy_P function copies a string from program space to a string in RAM ("buffer").
         Make sure your receiving string in RAM is large enough to hold whatever
         you are retrieving from program space. */


      for (int i = 0; i < 6; i++) {
        strcpy_P(buffer, (char *)pgm_read_ptr(&(string_table[i])));  // Necessary casts and dereferencing, just copy.
        Serial.println(buffer);
        delay(500);
      }
    }

If I don't want to use a string_table[] , then e.g. string_0 can be copied to buffer like this?

strcpy_P(buffer, pgm_read_ptr(string_0));

Thanks in advance.

2

What happened when you tried it?

3

The sketch compiled. :smiley:
I'm working on this without a board at the moment.

4

Looking in 'pgmspace.h', I see that strcpy_P() expects it's second argument to be const char *. But, pgm_read_ptr() returns void *. Technically, that's a type mismatch. But I think the rules are set so loose for AVR boards that it compiles. If it were my project, I'd cast to const char *.

5

Like this:

strcpy_P(buffer, (const char *)pgm_read_ptr(string_0));

A closer look at the example reminded me how basic my knowledge of pointers is. I'm totally lost at const char *const. :smiley: Or the const is only added so the table cannot be changed?

I know that whitespace does not matter, but wouldn't it be more readable here:

const char *const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

if the asterisk was placed immediately after char?

6

The two const have different meanings.

const char *const declares a point whose value is constant and which points to a const char. See:
https://stackoverflow.com/questions/1143262/what-is-the-difference-between-const-int-const-int-const-and-int-const

7

No, because string_0 is an array, not a pointer.
you'd want just:
strcpy_P(buffer, string_0);

8

Since it could also be no space around the asterisk

const char*const string_table[]

which we might all agree is the worst; or on both sides

const char * const string_table[]

If you're saying

const char* const string_table[]

is the best of the four, then maybe, but not by much. BTW, you could also express it as

char const * const string_table[]

which at least makes sense when read right-to-left:

  • "a const pointer": you cannot modify this variable
  • "to const char": you cannot modify the chars this points to

For whatever reason though, char const * is rare; it's usually const char *

The second arg to strcpy, regular or _P, is a const char *. When it is _P, it has to point inside program memory. But as a function argument, it's both a promise and a requirement.

  • "I the function will make a copy of the pointer"
  • "I require that it points to char"
  • "By making my copy point to const, I promise that I will not modify anything through this pointer. The compiler has already enforced this in my statements."

So of course you can pass const char *, which usually points to hard-coded strings, whether in PROGMEM or not. If you have "dog", you don't want some function to change that to "cat". (Although with PROGMEM, maybe you can't actually change it; don't know what happens when you try.)

The more interesting part is that you can also pass not-const, plain char *. You can create a buffer and fill it, then pass it off to a function that promises not to modify it.

The reverse is not true: if the function argument is plain char *, then it might modify the data there. That means you can't pass a const char *

pgm_read_ptr

  • takes a pointer to program memory to
  • read a pointer stored there
  • which it casts to void *, because it doesn't know what was being pointed to originally, so that you can then
  • static_cast to the appropriate pointer type

As already mentioned, you wouldn't pass string_0, because that's not a pointer to a pointer. (It's already a pointer to the thing you want.) When it does

that takes the address of the numbered array element. You might find it easier to understand by embracing a little "pointer math"

pgm_read_ptr(string_table + i)

(It definitely has fewer brackets and parentheses.) string_table is an array, and therefore also decays to a pointer. By itself, it is a pointer to the first element, [0]. With pointer math, adding the element number adds that multiple of the element size.

That example is about having a table of strings. Do you need a table?

9

I understand that string_0 is an array (const char), but as @gfvalvo pointed out: the strcpy_P() expects it's second argument to be const char *. Basicallypgm_read_ptr would get the void* pointer of the array and then we explicitly cast it to const char*.

Or is the string_0 in your version implicitly cast to const char*?

10

Thanks for the detailed answer. Those asterisk placements can be tricky, but reading the inserted stack overflow page and your answer helped me to understand them a little better.

No, the table is not needed, that's why my question came up.

11

Assuming you have no shortage of Flash memory, you have an interesting feature of the PROGMEM capability in the way it deals with trivial types and the fact that you'll likely use print() or write() to send the commands to your GSM shield and the fact that the print library knows how to read data from Flash directly, without an extra buffer.

The idea is to have a fixed size container for the commands, so you'll have to size it for the max command size (meaning that if you save "AT" it will be in a buffer large enough to hold also "AT+CMGS=\"+1234567890\"".

This is often OK because the flash memory is pretty large compared to RAM.

try this

constexpr uint8_t commandMaxSize = 30 + 1; // 30 characters plus trailing null

struct Command {
  char atCommand[commandMaxSize];
};

static const Command commands[] PROGMEM = {
  {"AT"},                       // Test the communication with the module
  {"AT+CSQ"},                   // Get the signal quality
  {"AT+CGMI"},                  // Get the manufacturer identification
  {"AT+CGMM"},                  // Get the model identification
  {"AT+CGSN"},                  // Get the IMEI number of the modem
  {"AT+CREG?"},                 // Check the network registration status
  {"AT+CMGF=1"},                // Set the SMS message format to text mode
  {"AT+CMGS=\"+1234567890\""},  // Send an SMS to a specified number
  {"AT+CIMI"},                  // Get the International Mobile Subscriber Identity (IMSI)
  {"AT+CSCS=\"GSM\""},          // Set the character set to GSM
};
const uint8_t nbCommands = sizeof commands / sizeof * commands;

void setup() {
  Serial.begin(115200); Serial.println();
  for (uint8_t i = 0; i < nbCommands; i++) {
    Serial.print(F("commands[")); Serial.print(i);
    Serial.print(F("]=\"")); Serial.print((__FlashStringHelper*) commands[i].atCommand);
    Serial.println(F("\""));
  }
}

void loop() {}

that will print

commands[0]="AT"
commands[1]="AT+CSQ"
commands[2]="AT+CGMI"
commands[3]="AT+CGMM"
commands[4]="AT+CGSN"
commands[5]="AT+CREG?"
commands[6]="AT+CMGF=1"
commands[7]="AT+CMGS="+1234567890""
commands[8]="AT+CIMI"
commands[9]="AT+CSCS="GSM""

you can add as many commands as you want to the array, that will increase the flash size accordingly but it won't change your RAM usage and you don't have to deal with strcpy_P() at all, just cast the pointer to a (__FlashStringHelper*) so that the right print() method is called (the one that knows how to extract the bytes from Flash).

Note that if you don't provide a large enough buffer (in my example if commandMaxSize is too small) the compiler will warn you with a message stating

warning: initializer-string for array of chars is too long [-fpermissive]
 };
 ^

so if you pay attention to warnings, you won't be in trouble.

now if you have an enum defined based on the commands of your array

enum : byte {AT, ATCSQ, ATCGMI, ATCGMM, ATCGSN, ATCREG, ATCMGF, ATCMGS, ATCIMI, ATCSCS};

then you can send a command to your GSM in a readable way like

gsmSerial.print((__FlashStringHelper*) commands[ATCGSN].atCommand);

and if you put that in a function

void sendCommand(byte commandID) {
  gsmSerial.print((__FlashStringHelper*) commands[commandID].atCommand);
}

then you can do just

sendCommand(ATCGSN);

which makes the code very readable.

12

This looks very neat, but it would require a complete refactor of my current code.

I'm using a sendATcommand() function to handle GSM module communication, for example:
sendATcommand("AT+FTPPUTPATH=\"/test/\"", "OK", 1000ul)
But now I realized that these passed char arrays are copied to the RAM.

So I'm planning just a quick fix, something like this:

const static char PATH_CMD[]   PROGMEM = "AT+FTPPUTPATH=";
const static char FTP_FOLDER[] PROGMEM = "\"/test/\"";

/*these will be used for many other commands as well inside the FTP function*/
char cmd_str[54] = "";
char aux_cmnd_str[18] = "";
char aux_data_str[36] = "";

/*this is just an example*/
strcpy_P(aux_cmnd_str, (const char*)pgm_read_ptr(PATH_CMD));
strcpy_P(aux_data_str, (const char*)pgm_read_ptr(FTP_FOLDER));

/*or simply just like this, it is not clear for me yet*/
// strcpy_P(aux_cmnd_str, PATH_CMD);
// strcpy_P(aux_data_str, FTP_FOLDER);   
snprintf(cmd_str, sizeof(cmd_str), "%s%s", aux_cmnd_str, aux_data_str);
sendATcommand(cmd_str, "OK", 1000ul);

But it's still not clear for me how to "strcpy_P" in this case.
The const static char arrays (e.g. PATH_CMD) will implicitly cast to const char*?

13

do you "own" the sendATcommand() function or is it in a library that expects a c-string in RAM ?

they are indeed const char* but the underlying pointer is an address in flash, not in RAM. so a function expecting a const char* will go seek the data in the wrong memory (RAM instead of Flash). that's what the (__FlashStringHelper*) cast in my previous code was meant for: you tell the print function "it's not really a const char* but another type and so the correct version of print() is called, the one reading the bytes from Flash)

14

I "have" it:

bool sendATcommand(const char* ATcommand, const char* expected_answer, const unsigned long timeout)
{  
  bool answer = false;
  uint8_t respx = 0;
  unsigned long previous = 0ul;

  /*initialize the string*/
  memset(response, '\0', RESPONSE_SIZE);

  /*timeout for while cycle*/
  previous = millis();

  while ( (Serial.available() > 0) &&
          ((millis() - previous) < timeout) )
  {
    /*Clean the input buffer*/
    Serial.read();
  }
  if (timeout != JUST_WAIT_FOR_IT)
  {
    /*Send the AT command*/
    Serial.println(ATcommand);
  }

  /*this loop waits for the answer with timeout check*/
  previous = millis();
  do
  {
    if (Serial.available() != 0)
    {
      /*if there is data in the UART input buffer, reads it and checks for the answer*/
      response[respx] = Serial.read();

      respx++;
      if (respx >= RESPONSE_SIZE)
      {
        respx = RESPONSE_SIZE - 1;
      }

      /*check if the desired answer is in the response of the module*/
      if (strstr(response, expected_answer) != NULL)
      {
        answer = true;
      }
    }
  }
  while ( !answer && ((millis() - previous) < timeout) );

  /*terminate the string, respx was limited to (RESPONSE_SIZE - 1)*/
  response[respx] = '\0';

  return answer;
}

I don't feel like following you totally.
The reason why my sendATcommand() calls use so much memory because I use this:

Serial.println(ATcommand);

and during "startup" all the passed commands are copied to RAM?

If I would simply change it to:

Serial.println(F(ATcommand));

the copy would not be necessary and my "guarded with fear" RAM would be spared?

(I cannot compile and test at the moment)

15

if you want to concatenate and use formatting strings, you can check this out

side note: test this

const char PATH_CMD[]   PROGMEM = "AT+FTPPUTPATH=";
const char FTP_FOLDER[] PROGMEM = "\"/test/\"";
const char CMD_FMT[] PROGMEM = "%S%S";

char cmdStr[100];

void setup() {
  Serial.begin(115200);
  snprintf_P(cmdStr, sizeof cmdStr, CMD_FMT, PATH_CMD, FTP_FOLDER);
  Serial.println(cmdStr);
}

void loop() {}

for your sendATcommand() code, you could consider ATcommand and expected_answer as being flash memory pointer and use the (__FlashStringHelper*) cast trick.

16

Could you give me an example of this?

17

something like this (typed here based on your function - so to be tested)

const char HELLO[] PROGMEM = "HELLO";
const char HI[] PROGMEM = "HI";
const char OK[] PROGMEM = "OK";
const char MONEY[] PROGMEM = "Do you want $100?";

constexpr uint8_t RESPONSE_SIZE = 50;
constexpr unsigned long JUST_WAIT_FOR_IT = 0;
char response[RESPONSE_SIZE + 1];


bool sendATcommand(const char* ATcommand, const char* expected_answer, const unsigned long timeout)
{
  bool answer = false;
  uint8_t respx = 0;
  unsigned long previous = 0ul;

  /*initialize the string*/
  memset(response, '\0', RESPONSE_SIZE);

  /*Clean the input buffer*/
  previous = millis();
  while ( (Serial.available() > 0) && ((millis() - previous) < timeout) ) Serial.read();

  if (timeout != JUST_WAIT_FOR_IT)
  {
    /*Send the AT command*/
    Serial.println((__FlashStringHelper*) ATcommand);
  }

  /*this loop waits for the answer with timeout check*/
  previous = millis();
  do {
    if (Serial.available() != 0)
    {
      /*if there is data in the UART input buffer, reads it and checks for the answer*/
      response[respx] = Serial.read();
      if (++respx >= RESPONSE_SIZE)respx = RESPONSE_SIZE - 1;
      /*check if the desired answer is in the response of the module*/
      answer = (strstr_P(response, expected_answer) != nullptr);
    }
  } while ( !answer && ((millis() - previous) < timeout) );

  /*terminate the string, respx was limited to (RESPONSE_SIZE - 1)*/
  response[respx] = '\0';

  return answer;
}

void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.println(); Serial.println();

  Serial.println(F("sending 'Hello', you'll have 5 seconds to answer with 'HI' in capital letters"));
  if (sendATcommand(HELLO, HI, 5000)) {
    Serial.println(F("Thanks, You are polite 😀"));
  } else {
    Serial.println(F("Duh! You are not polite 😱"));
  }

  Serial.println(); Serial.println();

  Serial.println(F("sending 'Do you want $100?', you'll have 5 seconds to answer with 'OK'"));
  if (sendATcommand(MONEY, OK, 5000)) {
    Serial.println(F("I wish I could generate money 😀"));
  } else {
    Serial.println(F("i see, you don't need the money!"));
  }
}
18

That was my bad. I got the datatypes correct, but the required level of indirection wrong. Sorry about that.

19

You can also use const __FlashStringHelper* as the type for the argument to the function, allowing you to use the F() macro in the function call instead of having to store the text in char arrays separately.

constexpr uint8_t RESPONSE_SIZE = 50;
constexpr unsigned long JUST_WAIT_FOR_IT = 0;
char response[RESPONSE_SIZE + 1];


bool sendATcommand(const __FlashStringHelper* ATcommand, const __FlashStringHelper* expected_answer, const unsigned long timeout)
{
  bool answer = false;
  uint8_t respx = 0;
  unsigned long previous = 0ul;

  /*initialize the string*/
  memset(response, '\0', RESPONSE_SIZE);

  /*Clean the input buffer*/
  previous = millis();
  while ( (Serial.available() > 0) && ((millis() - previous) < timeout) ) Serial.read();

  if (timeout != JUST_WAIT_FOR_IT)
  {
    /*Send the AT command*/
    Serial.println( ATcommand);
  }

  /*this loop waits for the answer with timeout check*/
  previous = millis();
  do {
    if (Serial.available() != 0)
    {
      /*if there is data in the UART input buffer, reads it and checks for the answer*/
      response[respx] = Serial.read();
      if (++respx >= RESPONSE_SIZE)respx = RESPONSE_SIZE - 1;
      /*check if the desired answer is in the response of the module*/
      answer = (strstr_P(response, (const char*)expected_answer) != nullptr);
    }
  } while ( !answer && ((millis() - previous) < timeout) );

  /*terminate the string, respx was limited to (RESPONSE_SIZE - 1)*/
  response[respx] = '\0';

  return answer;
}

void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.println(); Serial.println();

  Serial.println(F("sending 'Hello', you'll have 5 seconds to answer with 'HI' in capital letters"));
  if (sendATcommand(F("HELLO"), F("HI"), 5000)) {
    Serial.println(F("Thanks, You are polite 😀"));
  } else {
    Serial.println(F("Duh! You are not polite 😱"));
  }

  Serial.println(); Serial.println();

  Serial.println(F("sending 'Do you want $100?', you'll have 5 seconds to answer with 'OK'"));
  if (sendATcommand(F("Do you want $100?"), F("OK"), 5000)) {
    Serial.println(F("I wish I could generate money 😀"));
  } else {
    Serial.println(F("i see, you don't need the money!"));
  }
}
20

@J-M-L , @david_2018 thanks for the detailed answers!

This would mean the least modification required for the code, this could really work?!