Solution of reducing the SRAM used by large string array with a switch-case

Dear Arduino community,

I have 30.720 Bytes Flash and 2.048 Bytes SRAM on my ATMEGA.
At the top of my code, I defined a large string array:

String LargeVariable[200] = {"foo","bar", ... };

This large string array consumed terrible amounts of my SRAM (up to 107%). Obviously, the program didn't run.

Then I did some reaserch on the web and found an accepted solution at stackexchange:
http://arduino.stackexchange.com/questions/221/what-can-i-do-if-i-run-out-of-flash-memory-or-sram

Basically it say that one should not use large global variables. Instead variables should be made local so that the memory can be freed up after usage.

So instead I wrote a function that used a case-switch method that I can use to access the array element I need:

String get_LargeVariable(int pos) {
  switch (pos) {
    case 0: return "foo"; break;
    case 1: return "bar"; break;
    case 2: 
    ...
    case 200: 
  }
}

// get element with
get_LargeVariable(5);

Now I have no global variable any more as before. I should have freed up some SRAM. But the SRAM is still used up by 107%.
Does anybody know what I am doing wrong?

Thank you very much,
MegaMatze

Long copy of the mentioned post at stackexchange:

Runtime data (SRAM) optimisation tends to be a bit easier when you're used to it. A very common pitfall for beginner programmers is using too much global data. Anything declared at global scope will exist for the entire lifetime of the sketch, and that isn't always necessary. If a variable is only used inside one function, and it doesn't need to persist between calls, then make it a local variable. If a value needs to be shared between functions, consider if you can pass it as a parameter instead of making it global. That way you'll only use SRAM for those variables when you actually need it.

Another killer for SRAM usage is text processing (e.g. using the String class). Generally speaking, you should avoid doing String operations if possible. They are massive memory hogs. For example, if you're outputting lots of text to serial, use multiple calls to Serial.print() instead of using string concatenation. Also try to reduce the number of string literals in your code if possible.

Don't use RAM for constant strings, use PROGMEM.

Great!

Using
const dataType variableName[] PROGMEM = {data0, data1, data3...};
to define constant arrays saves about 30% of the SRAM that the array consumes.

Is it possible to free up even more? I would like to increase the size of my array.

O wait. You mean this?

#include <avr/pgmspace.h>
const char string_0[] PROGMEM = "String 0";   // "String 0" etc are strings to store - change to suit.
const char string_1[] PROGMEM = "String 1";
const char string_2[] PROGMEM = "String 2";
const char string_3[] PROGMEM = "String 3";
const char string_4[] PROGMEM = "String 4";
const char string_5[] PROGMEM = "String 5";


// Then set up a table to refer to your strings.

const char* const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

Does this method save even more SRAM?

At the top of my code, I defined a large string array:

Bullshit. That is an array of Strings, not strings. NOT the same thing. At all.

Post ALL of your code.

const char* const string_table[] PROGMEM = {string_0, string_1, string_2, string_3, string_4, string_5};

You don't need "PROGMEM". This stores the array in program memory as well.

const char* const string_table[] = {string_0, string_1, string_2, string_3, string_4, string_5};

Works if you don't need to modify the table content.

"const" is NOT sufficient to cause data to be stored in flash on AVR CPUs.
(it IS on ARM chips...)

At the top of my code, I defined a large string array:

Why?

Read this: Putting constant data into program memory (PROGMEM)

String get_LargeVariable(int pos) {

Forget the String class if you want to be memory-efficient.

"const" is NOT sufficient to cause data to be stored in flash on AVR CPUs.

Correct.
But works in this case (I learned it from Nick Gammon and checked with the avr mem tool) :slight_smile:

First of all thank you very much for all that answers.
I try to summerize to get the essence out of it.

First, change String to char.
Additionally PROGMEM ist not necessary.
And making the array of strings constant does not change the SRAM usage.

Test:

const String MyVar[96] PROGMEM = {"blabla", "blub", "sjdvbhcuze", ... };

PROGMEM: 20.010 Bytes (65%)
SRAM: 2.297 Bytes (112%)

const char* const MyVar[96] = {"blabla", "blub", "sjdvbhcuze", ... };

PROGMEM: 17.306 Bytes (56%)
SRAM: 1.341 Bytes (65%)

So that already solves a large part of the problem.

But when I access it like so:

MyVar[1]

What exactly do I get? An array os chars?

char test[ ] = MyVar[1];

Following Nicks link, there are a lot of examples of which this one seems to be the best way:

const int NUMBER_OF_ELEMENTS = 10;
const int MAX_SIZE = 12;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = { 
 { "Furnace on" }, 
 { "Furnace off" }, 
 { "Set clock" }, 
 { "Pump on" }, 
 { "Pump off" }, 
 { "Password:" }, 
 { "Accepted" }, 
 { "Rejected" }, 
 { "Fault" }, 
 { "Service rqd" }, 
 };

I seems to me that this is the same as in my examply above. It simply avoids the string data type by using char instead.

Or is there any further significant advantage in SRAM usage?

It simply avoids the string data type by using char instead.

Sp. "String", which also incurs extra RAM use by each object instantiated, over and above the string content.
PROGMEM is the way to go.
Oh wait, I already wrote that.

Sorry to bother you with that again, but florinc said that...

florinc:
You don't need "PROGMEM". This stores the array in program memory as well.

const char* const string_table[] = {string_0, string_1, string_2, string_3, string_4, string_5};

That stores an index to the string array, in RAM.

florinc:

"const" is NOT sufficient to cause data to be stored in flash on AVR CPUs.

Correct.
But works in this case (I learned it from Nick Gammon and checked with the avr mem tool) :slight_smile:

I never said that. On the AVR chips you specifically have to use the PROGMEM attribute. For non-arrays, making things const will help, yes, because the compiler doesn't have to create a variable in RAM. For an array though, with pointers to it, the compiler assumes pointers are to RAM, so it copies the entire array into RAM to make this true.

Now I think I got it.

And in this solution, everything is stored in PROGMEM? Not only pointers or something like that?

const int NUMBER_OF_ELEMENTS = 10;
const int MAX_SIZE = 12;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
 { "Furnace on" },
 { "Furnace off" },
 { "Set clock" },
 { "Pump on" },
 { "Pump off" },
 { "Password:" },
 { "Accepted" },
 { "Rejected" },
 { "Fault" },
 { "Service rqd" },
 };

And I can access it just like so:

char test = descriptions[2];

It puts the lot in PROGMEM, where it belongs.

And I can access it just like so

Err, no.

No, you can't pull something from PROGMEM without using an accessing function. Try it and see.

Here is an example of accessing by making a copy of one item:

const int NUMBER_OF_ELEMENTS = 10;
const int MAX_SIZE = 12;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
 { "Furnace on" },
 { "Furnace off" },
 { "Set clock" },
 { "Pump on" },
 { "Pump off" },
 { "Password:" },
 { "Accepted" },
 { "Rejected" },
 { "Fault" },
 { "Service rqd" },
 };

void setup() 
  {
  Serial.begin (115200); 
  for (int i = 0; i < 10; i++)
    {
    char buf [MAX_SIZE];
    strcpy_P (buf, (const char *) &descriptions [i]);
    Serial.println (buf);
    }
  }

void loop() 
{
}

WOW! That works like a charm. According to my first tests it uses massivly less memory. Crazy!

But what I don't understand is, why my solution in the first post didn't work. I removed the long array of strings and put everything into a switch-case function. So there was no global variable and almost everything is put into programcode and should be stored in PROGMEM.

So why does it need exactly as much SRAM as the original array of strings?

So why does it need exactly as much SRAM as the original array of strings?

Because RAM is the natural habitat of the string.

MegaMatze2:
Now I think I got it.

And in this solution, everything is stored in PROGMEM? Not only pointers or something like that?

const int NUMBER_OF_ELEMENTS = 10;

const int MAX_SIZE = 12;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
{ "Furnace on" },
{ "Furnace off" },
{ "Set clock" },
{ "Pump on" },
{ "Pump off" },
{ "Password:" },
{ "Accepted" },
{ "Rejected" },
{ "Fault" },
{ "Service rqd" },
};



[
And I can access it just like so:


char test = descriptions[2];

no, descriptions[] are in program space, AVR's are harvard Architecture, Program space and Ram Space are different.
to transfer from program space to ram space you have to copy each byte from Program space to a ram buffer before you can use it.

#include <avr/pgmspace.h>


const int NUMBER_OF_ELEMENTS = 10;
const int MAX_SIZE = 12;

const char descriptions [NUMBER_OF_ELEMENTS] [MAX_SIZE] PROGMEM = {
 { "Furnace on" },
 { "Furnace off" },
 { "Set clock" },
 { "Pump on" },
 { "Pump off" },
 { "Password:" },
 { "Accepted" },
 { "Rejected" },
 { "Fault" },
 { "Service rqd" },
 };

/* function to transfer progmem to RAM */

void getMsg(char msg[],uint8_t msgnum){
for(uint8_t i=0;i<MAX_SIZE;i++){
  msg[i] = pgm_read_byte_near(descriptions[msgnum] + i);
  }
} 

/* **********************  a more compact way to store message ********* */
const char packedDes[] PROGMEM = "UnKnown MSG\000" "Furnace on\000" "Furnace off\000" "Set clock\000" 
  "Pump on\000" "Pump off\000" "Password:\000" "Accepted\000" "Rejected\000" "Fault\000"
 "Service rqd\000\000";

uint16_t findMsg(uint16_t msgnum){
uint16_t result=0;
bool done=false, oneZero=false;
while(!done&&msgnum>0){
 if(pgm_read_byte_near(packedDes+result++)==0){ //string end marker
 done=oneZero; // double zero marks end of message table
 msgnum--;
 oneZero=true;
 }
 else oneZero=false;
}
if(done) result = 0; // ran into double zero, at end of list, msgnum > max messages
return result;
}

void getPackedMsg(char msg[],uint16_t msgnum){
uint16_t msgOffset = findMsg(msgnum);
uint8_t i=0;
do{
  msg[i] = pgm_read_byte_near(packedDes + msgOffset+i++);
  }while(msg[i-1]!=0);
}

uint16_t countMsgs(){
uint16_t result=0,i=0;
bool done=false, oneZero=false;
while(!done){
 if(pgm_read_byte_near(packedDes+i++)==0){
 done=oneZero;
 if(!done)result++;
 oneZero=true;
 }
 else oneZero=false;
}
return result;
}

/* *************************** */

char descriptionBuffer[MAX_SIZE]; // 12

void setup(){
Serial.begin(9600);
Serial.println(" accessing PRGMMem array structure ");
for(uint8_t i = 0; i < NUMBER_OF_ELEMENTS;i++){
  getMsg(descriptionBuffer,i);
 Serial.print('|');
  Serial.print(descriptionBuffer);
 Serial.println('|');
  }
Serial.println("now accessing packed messages, a little slower");
uint16_t msgcnt = countMsgs();
do{
  getPackedMsg(descriptionBuffer,--msgcnt); //msgcount = # of messages, msgnum 0..msgcnt-1
 Serial.print('|');
  Serial.print(descriptionBuffer);
 Serial.println('|');
  }while(msgcnt>0);
 
}

void loop(){}

here is the output

 accessing PRGMMem array structure 
|Furnace on|
|Furnace off|
|Set clock|
|Pump on|
|Pump off|
|Password:|
|Accepted|
|Rejected|
|Fault|
|Service rqd|
now accessing packed messages, a little slower
|Service rqd|
|Fault|
|Rejected|
|Accepted|
|Password:|
|Pump off|
|Pump on|
|Set clock|
|Furnace off|
|Furnace on|
|UnKnown MSG|

Chuck.