PROGMEM and string retrieval

Projects Programming
1

Hi all,

I'm hoping someone can help here as I've been stuck on this for some time. My Arduino Nano has run out of SRAM space and I'm trying to move some string constants to program memory / flash from RAM.

I want to store things like wifi ssid, password, API keys, server name (for ThingSpeak) etc in flash and access them locally in the function vs being globally defined.

I've been testing the standard sketch attached which works fine, but how do I call two or more of the strings consecutively to use in my function. A single string works fine, but I need to call for example

status = WiFi.begin(ssid,pwd);

where both ssid and pwd are stored via PROGMEM. Inserting the strcpy_P into the WiFi.begin() does not work as the string is in the buffer which is overwritten when the next string is referenced.

#include <avr/pgmspace.h>
const char string_0[] PROGMEM = "String 0 Test";   // "String 0" etc are strings to store - change to suit.
const char string_1[] PROGMEM = "String 1 Test";
const char string_2[] PROGMEM = "String 2 Test";
const char string_3[] PROGMEM = "String 3 Test";
const char string_4[] PROGMEM = "String 4 Test";
const char string_5[] PROGMEM = "String 5 Test";

// Then set up a table to refer to your strings.

const char * const string_table[] PROGMEM =     // change "string_table" name to suit
{   
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  string_5 };

char buffer[15];    // make sure this is large enough for the largest string it must hold

void setup()        
{
  Serial.begin(9600);
  
  strcpy_P(buffer, (char*)pgm_read_word(&(string_table[1])));
  Serial.println( buffer );
 
}

void loop()       
{
  //Not used
}

So I need to be able to so something like,

status = WiFi.begin(string_0,string_1);

Any help greatly appreciated.

1 Like
2

Maybe that's what you're looking for: how to convert char * to string

3

Arduino > Reference > Language > Variables > Utilities > Progmem

4

I tried to perform the compilation, but it gave error, it seems that the library does not accept the type of variable.

#include <avr/pgmspace.h>
#include <WiFi.h>

//SSID of your network
const char ssid[] PROGMEM = "yourNetwork";
//password of your WPA Network
char pass[] = "secretPassword";

void setup()
{
 WiFi.begin(ssid, pass);
}

void loop () {}

sketch_mar18a:11:23: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]

WiFi.begin(ssid, pass);

^

In file included from C:\Users\robso\AppData\Local\Temp\arduino_modified_sketch_607470\sketch_mar18a.ino:2:0:

C:\Arduino IDE\arduino-1.8.12-windows\arduino-1.8.12\libraries\WiFi\src/WiFi.h:79:9: error: initializing argument 1 of 'int WiFiClass::begin(char*, const char*)' [-fpermissive]

int begin(char* ssid, const char *passphrase);

^

Using library WiFi at version 1.2.7 in folder: C:\Arduino IDE\arduino-1.8.12-windows\arduino-1.8.12\libraries\WiFi
Using library SPI at version 1.0 in folder: C:\Arduino IDE\arduino-1.8.12-windows\arduino-1.8.12\portable\packages\esp8266\hardware\esp8266\2.6.3\libraries\SPI
exit status 1
invalid conversion from 'const char*' to 'char*' [-fpermissive]

int begin(char* ssid, const char *passphrase);

5

Maybe using a buffer:

#include <avr/pgmspace.h>
#include <WiFi.h>

//SSID of your network
const char ssid1[] PROGMEM = "yourNetwork1";
const char ssid2[] PROGMEM = "yourNetwork2";
const char ssid3[] PROGMEM = "yourNetwork3";
const char ssid4[] PROGMEM = "yourNetwork4";
//password of your WPA Network
char pass[] = "secretPassword";


void setup()
{
  char charBuf[sizeof(ssid1)];

  memcpy(charBuf, ssid1, sizeof ssid1);
  
  WiFi.begin(charBuf, pass);
}

void loop () {}

http://www.cplusplus.com/reference/cstring/memcpy/

6

WiFi.begin() is not set up to handle PROGMEM directly, so the only way may be to copy the strings to separate buffers, which defeats the purpose of storing them in PROGMEM to begin with.

rtek1000:
Maybe that's what you're looking for: how to convert char * to string

Despite the title, that is talking about Strings, which you DO NOT want to use on a Nano, especially if you are low on SRAM to begin with.

Incidentally, you do not need to use strcpy_P to copy the string to a buffer if you are using it for a print() statement, print() is quite capable of handling a string stored in PROGMEM, as long as you cast it to __FlashStringHelper*

  strcpy_P(buffer, (char*)pgm_read_word(&(string_table[1])));
  Serial.println( buffer );

// this can be replaced with the following:

  Serial.println((__FlashStringHelper*)pgm_read_word(&string_table[1]));
7

WiFi.begin() is not set up to handle PROGMEM directly, so the only way may be to copy the strings to separate buffers, which defeats the purpose of storing them in PROGMEM to begin with.

Thanks, I'll give that idea up then. :expressionless:
I'm using a ESP8266-01 over Softserial as a Wifi shield to send weather data.

I've already added the F macro everywhere for Serial.print and for things like client.print(F("Host: api.thingspeak.com\n"));

I'm currently close to maxed out on the Nano, I've determined that at exactly 77% dynamic memory utilization, program gets unstable.

Current use:

Sketch uses 29412 bytes (95%) of program storage space. Maximum is 30720 bytes.
Global variables use 1467 bytes (71%) of dynamic memory, leaving 581 bytes for local variables. Maximum is 2048 bytes.

I was hoping to add Blynk and some additional network client support, but looks like I may have to give that up for now. At least ThingSpeak is working reliably.

8

Wouldn't it be possible to use the ESP8266 to perform the operations autonomously, and just command the Arduino to do small things?

9

rtek1000:
Arduino > Reference > Language > Variables > Utilities > Progmem

Thanks, I've looked through these examples already, but as someone else pointed out, WiFi.begin() in <WiFiEsp.h> doesn't seem to support this unfortunately.

I've tried using the strcpy_P(buffer, (char*)pgm_read_word(&(string_table[1]))) within the WiFi.begin() function which of course does not work.

10

rtek1000:
Wouldn't it be possible to use the ESP8266 to perform the operations autonomously, and just command the Arduino to do small things?

Thanks, I have Wemos D1 Mini that would be better suited sounds like, but as I'd have to refactor all the code from the Nano, I'm reticent to go that route just at the moment. In any case, the point of the project was for me to learn Arduino platform and software. :slight_smile:

So far I have the following working:

  • Nano receiving BME280 remote data over a RF433MHz link (Remote transmitter is also another Nano);
  • Reading local data from a DHT22;
  • Local RTC with DS3231;
  • Printing all local/remote data to 128x64 Oled;
  • Connecting to ESP8266-01 over Softserial;
  • Sending all weather data to ThingSpeak;

Hence why my code is a bit bloated....thanks to everyone for all the responses btw, it's fantastic!

11

rtek1000:
Maybe using a buffer:

#include <avr/pgmspace.h>

#include <WiFi.h>

//SSID of your network
const char ssid1[] PROGMEM = "yourNetwork1";
const char ssid2[] PROGMEM = "yourNetwork2";
const char ssid3[] PROGMEM = "yourNetwork3";
const char ssid4[] PROGMEM = "yourNetwork4";
//password of your WPA Network
char pass[] = "secretPassword";

void setup()
{
 char charBuf[sizeof(ssid1)];

memcpy(charBuf, ssid1, sizeof ssid1);
 
 WiFi.begin(charBuf, pass);
}

void loop () {}




http://www.cplusplus.com/reference/cstring/memcpy/

Thanks much, but wouldn't the character buffer take about the same space as storing the ssid directly in SRAM? Do you think this memory be released back to the heap once used by the WiFi function and setup() is complete? I'll try give this a go, thanks!

12

tsarath:
Thanks much, but wouldn't the character buffer take about the same space as storing the ssid directly in SRAM? Do you think this memory be released back to the heap once used by the WiFi function and setup() is complete? I'll try give this a go, thanks!

So I gave it a try. It seems to use exactly the same amount of RAM as having it defined directly in memory as you'd expect, oh well.. :expressionless:

Here is the code I tried for reference which uses 22% (459 bytes) of the RAM on the Nano!!

#include <avr/pgmspace.h>
#include <WiFiEsp.h> 

#ifndef HAVE_HWSERIAL1
#include "SoftwareSerial.h"   // Include softserial for passing data to ESP
SoftwareSerial Serial1(5,4);  // (D5/D4) on Nano for (RX,TX) for communication with ESP8266-01
#endif

// const char* ssid = "things";
// const char* pass = "MyPassword";

int status = WL_IDLE_STATUS;     // the Wifi radio's status

//SSID of your network
const char ssid1[] PROGMEM = "things";

//password of your WPA Network
char pass[] = "MyPassword";


void setup()
{
  char charBuf[sizeof(ssid1)];

  memcpy(charBuf, ssid1, sizeof ssid1);
  
  Serial.begin(9600);       //  Setup Serial Monitor
  while(!Serial);           //  time to get serial running
      Serial.println(F("WeatherBase Rx Init.."));
    
  Serial1.begin(9600);      //  Setup softserial to ESP8266-01
  WiFi.init(&Serial1);
   
   while ( status != WL_CONNECTED) {                  // attempt to connect to WiFi network
    status = WiFi.begin(charBuf, pass);               // Connect to WPA/WPA2 network
    delay(10000);                                     // Wait 10 seconds for connection:
    }
  Serial.println(F("Connected!"));
 
}

void loop () {}

It's probably safe to say that it's maxed out as is at this point...

13

david_2018:
WiFi.begin() is not set up to handle PROGMEM directly, so the only way may be to copy the strings to separate buffers, which defeats the purpose of storing them in PROGMEM to begin with.

Despite the title, that is talking about Strings, which you DO NOT want to use on a Nano, especially if you are low on SRAM to begin with.

Incidentally, you do not need to use strcpy_P to copy the string to a buffer if you are using it for a print() statement, print() is quite capable of handling a string stored in PROGMEM, as long as you cast it to __FlashStringHelper*

  strcpy_P(buffer, (char*)pgm_read_word(&(string_table[1])));

Serial.println( buffer );

// this can be replaced with the following:

Serial.println((__FlashStringHelper*)pgm_read_word(&string_table[1]));

A quick follow-up question on the above suggestion which I got working fine standalone. However when I tried to use it in the following additional function calls as which I believe are part of the WiFiEsp.h, this ended up not working either. :confused:

I tried replacing server and apiKey in the code below with (__FlashStringHelper*)pgm_read_word(&string_table[1]) equivalent, would this be doable you think? apiKey and server are globally defined String and const char currently.

#include <WiFiEsp.h> 

String apiKey = "5J41ZND8HXXXXXX";
const char* server = "api.thingspeak.com";

...

void updateThingSpeak(){

....

 if (client.connect(server, 80)) {
      String postStr = apiKey;
      postStr +="&field1=";
      postStr += String(localtemp);
      postStr +="&field2=";
      postStr += String(localhumid);
      postStr +="&field3=";
      postStr += String(bmeData.temp);
      postStr +="&field4=";
      postStr += String(bmeData.hum);
      postStr +="&field5=";
      postStr += String(bmeData.pres);
      postStr += "\r\n\r\n";
     
      
      client.print(F("POST /update HTTP/1.1\n"));
      client.print(F("Host: api.thingspeak.com\n"));
      client.print(F("Connection: close\n"));
      client.print("X-THINGSPEAKAPIKEY: " + apiKey + "\n");
      client.print(F("Content-Type: application/x-www-form-urlencoded\n"));
      client.print(F("Content-Length: "));
      client.print(postStr.length());
      client.print(F("\n\n"));
      client.print(postStr);
      lastConnectionTime = millis();
....

}

Many thanks for any help.

14

Can you post the entire sketch? The last code you posted can be done with less ram usage (an no use of Strings), but really need to know data types of the variable to give a good example.

Avoid the String data type, it can corrupt memory, and you do not have the ram needed to use it.

15

david_2018:
Can you post the entire sketch? The last code you posted can be done with less ram usage (an no use of Strings), but really need to know data types of the variable to give a good example.

Avoid the String data type, it can corrupt memory, and you do not have the ram needed to use it.

Sure, attached. Sorry for all the commented out sections, I've been trying various things out and haven't cleaned up the code as much as I would like.

This version uses the following on the Nano:

Sketch uses 29348 bytes (95%) of program storage space. Maximum is 30720 bytes.
Global variables use 1467 bytes (71%) of dynamic memory, leaving 581 bytes for local variables. Maximum is 2048 bytes

Any additional suggestions to drive down memory utilization (and flash storage) would be greatly welcome. Only additional thing I'd like to add to this sketch is Blynk support in addition to ThingSpeak but seems to have maxed our here without additional optimizations.

Thanks

WeatherBase_v8c.ino (15 KB)

16

In reference to rtek1000's suggestion of using a buffer for the ssid and pwd, that will work, as long as there is sufficient ram to create the buffers. The buffers are local to setup(), so any ram used is freed at the end of setup(). Might be better to place the code inside its own function, then call that from within setup(), so that the ram for the buffers is only needed while executing the WiFi.begin() statement and freed when returning to setup().

const char ssid[] PROGMEM = "things";
const char pwd[] PROGMEM = "MyPassword";

//this code is placed inside setup()
  char ssidbuff[strlen_P(ssid) + 1]; //buffer for ssid
  char pwdbuff[strlen_P(pwd) + 1]; //buffer for pwd
  strcpy_P(ssidbuff, ssid);
  strcpy_P(pwdbuff, pwd);
  WiFi.init(&Serial1);
  while ( status != WL_CONNECTED) {                 // attempt to connect to WiFi network
    status = WiFi.begin(ssidbuff, pwdbuff);                 // Connect to WPA/WPA2 network
    delay(10000);
  }

As for the updateThingSpeak function, I'm not familiar with the WiFiEsp library, or whether the data has to be printed in a single .print() statement or can be split amount multiple .print() statements. You can try this code and see if it will work. The buff[] array used in the sub functions is overly large, but I'm not sure of the maximum size of your data.

const char apiKey[] PROGMEM = "5J41ZND8XXXXXXXX";

void updateThingSpeak() {           // Send data over to ESP8266-01
  /*
     // Print Update Response to Serial Monitor
    if (client.available()) {
      char c = client.read();
      Serial.print(c);
    }

      // Disconnect from ThingSpeak
    if (!client.connected() && lastConnected) {
      Serial.println(F("...disconnected"));
      Serial.println();
      client.stop();
    }
  */
  // Update ThingSpeak
  if (!client.connected() && (millis() - lastConnectionTime > updateThingSpeakInterval)) {
    lastConnected = client.connected();
    if (client.connect(server, 80)) {
      client.print(F("POST /update HTTP/1.1\n"));
      client.print(F("Host: api.thingspeak.com\n"));
      client.print(F("Connection: close\n"));
      client.print(F("X-THINGSPEAKAPIKEY: "));
      client.print((__FlashStringHelper*)apiKey);
      client.print(F("\n"));
      client.print(F("Content-Type: application/x-www-form-urlencoded\n"));
      client.print(F("Content-Length: "));
      client.print(postStrLen());
      client.print(F("\n\n"));
      postStrPrint(client);

      lastConnectionTime = millis();

      if (client.connected()) {
        Serial.println(F("Connecting..."));
        postStrPrint(Serial);
      }
    }
  }
  client.flush();
  client.stop();
}

int postStrLen() {
  char buff[16];
  return ( strlen_P(apiKey) +
           8 + //strlen("&field1=");
           strlen(dtostrf(localtemp, 4, 2, buff)) +
           8 + //strlen("&field2=");
           strlen(dtostrf(localhumid, 4, 2, buff)) +
           8 + //strlen("&field3=");
           strlen(dtostrf(bmeData.temp, 4, 2, buff)) +
           8 + //strlen("&field4=");
           strlen(dtostrf(bmeData.hum, 4, 2, buff)) +
           8 + //strlen("&field5=");
           strlen(dtostrf(bmeData.pres, 4, 2, buff)) +
           4 //strlen("/r/n/r/n");
         );
}

void postStrPrint(Stream &port) {
  char buff[16];
  port.print((__FlashStringHelper*)apiKey);
  port.print(F("&field1="));
  port.print(dtostrf(localtemp, 4, 2, buff));
  port.print(F("&field2="));
  port.print(dtostrf(localhumid, 4, 2, buff));
  port.print(F("&field3="));
  port.print(dtostrf(bmeData.temp, 4, 2, buff));
  port.print(F("&field4="));
  port.print(dtostrf(bmeData.hum, 4, 2, buff));
  port.print(F("&field5="));
  port.print(dtostrf(bmeData.pres, 4, 2, buff));
  port.print(F("\r\n\r\n"));
}
17

david_2018:
In reference to rtek1000's suggestion of using a buffer for the ssid and pwd, that will work, as long as there is sufficient ram to create the buffers. The buffers are local to setup(), so any ram used is freed at the end of setup(). Might be better to place the code inside its own function, then call that from within setup(), so that the ram for the buffers is only needed while executing the WiFi.begin() statement and freed when returning to setup().

Thanks much, will give the suggestions a try and let you know.

18

tsarath:
Thanks much, will give the suggestions a try and let you know.

Hi again, just wanted to let you know the suggested changes seems to work well for both the WiFi.begin() and updateThingSpeak() routines.

Both memory and flash usage went down a quite a bit, the sketch now uses the following resources below.

Sketch uses 28536 bytes (92%) of program storage space. Maximum is 30720 bytes.
Global variables use 1323 bytes (64%) of dynamic memory, leaving 725 bytes for local variables. Maximum is 2048 bytes.

This should hopefully give me enough headroom to add Blynk support now as well. Thanks a lot for your help, much appreciated! :slight_smile:

19

You do need to be careful with the dynamic memory reported by the compiler, it does not count memory temporarily allocated inside functions.

I did find a few other things that will reduce your memory usage. The compiler will normally notice when you have identical string literals in the print() statements and only store a single string in memory, directing all references to that single string. Unfortunately it does not do that when using the F() parameter, so you can end up with duplicate strings stored in program memory. You can save a few bytes by declaring the string yourself and printing it instead of using literals.

If you run out of program storage space, and have access to an ISP programmer (or another arduino to use as an ISP programmer), you can program the Nano with the Uno bootloader. That will give you an additional 1536 bytes of program storage.

Attached is your sketch with a few changes to get the memory usage a bit lower (I didn't notice before that the server string could also be stored in PROGMEM).

WeatherBase_v8c_test.ino (16.9 KB)

20

david_2018:
You do need to be careful with the dynamic memory reported by the compiler, it does not count memory temporarily allocated inside functions.

I did find a few other things that will reduce your memory usage. The compiler will normally notice when you have identical string literals in the print() statements and only store a single string in memory, directing all references to that single string. Unfortunately it does not do that when using the F() parameter, so you can end up with duplicate strings stored in program memory. You can save a few bytes by declaring the string yourself and printing it instead of using literals.

If you run out of program storage space, and have access to an ISP programmer (or another arduino to use as an ISP programmer), you can program the Nano with the Uno bootloader. That will give you an additional 1536 bytes of program storage.

Attached is your sketch with a few changes to get the memory usage a bit lower (I didn't notice before that the server string could also be stored in PROGMEM).

Thanks much for the advice, I'll update, will look into reducing the String print lines or using similar messages as you suggest.

Unfortunately, there seems to be an unintended consequence of the additional postStrLen() and postStrPrint() functions. From what I can tell, the Nano spends more time executing updateThingSpeak() function than before and as a result rf_Receive() no longer works. I have verified if I comment out updateThingSpeak() things work as before.

rf_Receive() is asynchronous and I think is not getting executed correctly as a result even-though there has been no change in the code to that function, and hence does not receive the remote data over the RF link. :o

I've used delay commands as you can see in the main loop to step through each sequence. I tried increasing the updateThingSpeak() period to every 45s or 60s, but this didn't seem to help for some reason.

Not sure if you have any suggestions here, I'll try to debug this later today. I've tried some scheduler code before, but couldn't get it to work reliably in place of the delay functions in the loop whereto Nano is essentially doing nothing. I know some other Weather station code examples I have uses ISRs etc, but hopefully it will be a simpler fix.