SRAM cost of using PROGMEM

Using Arduino Programming Questions
1

Hello Arduino-Gurus,

short version: If I put a value in Flash using PROGMEM, does this require space in SRAM as well, e.g., for a pointer to the memory address in Flash?

long version:
I have written a sketch that is running out of SRAM, therefore I need to cut down on data stored in SRAM. I have several values that are currently defined as variables but actually have a fixed value that is known at development time.
I am now thinking about moving these values to Flash using PROGMEM. However, I am not sure whether I will actually save anything as I am dealing with a bunch of uint16_t here.

Looking at this problem globally, I would expect that since when I use PROGMEM, everything is known at compile time, the calls to copy data from the PROGMEM'd variable into SRAM for processing would amount to a bunch of MOV (or LOAD or whatever the AVR equivalent is) instructions with a fixed operand address, which also end up in Flash, i.e., zero SRAM usage. However, a number of websites that (vaguely!) discuss PROGMEM indicate that a pointer, stored in SRAM, is involved. For strings this clearly saves memory, I'm just wondering whether I'll replace one memory word containing the value by another memory word containing a pointer to the variable and thus making matters worse.

Regards,
deltaphi

2

I'm just wondering whether I'll replace one memory word containing the value by another memory word containing a pointer to the variable and thus making matters worse.

It won't make matters worse, but it might not help.

Can you put the values in an array, and store the array in PROGMEM?
Since the value doesn't change, are the variables declared const?

3

Hello,

It won't make matters worse, but it might not help.

Can you please elaborate on this or point me to a website with a proper technical discussion of how PROGMEM is implemented?

The values are not declared const. Currently they are (regular) member variables of classes that are instantiated in setup() - but that is a battle I have to fight with another developer.

Would making them const help? Note that I need to use external linkage for these values. They are sketch-specific, i.e., every sketch has to use different values, however their actual use happens inside a library.

I contemplated other options for freeing up SRAM, but they do not seem really sensible if there is ever anyone besides me using this code.
Option 1: PROGMEM'ing a struct. Similar to the suggested Array method, but quite a hassle for a user, since depending on the use of the library, not all values are present in every sketch.
Option 2: Using an "external (weak) getValueWhatever()" method. This should truely come at zero (static) SRAM cost but is even harder to understand for other developers.

Regards,
deltaphi

4

Currently they are (regular) member variables of classes

Then you can't put them in PROGMEM.

Note that I need to use external linkage for these values. They are sketch-specific, i.e., every sketch has to use different values, however their actual use happens inside a library.

Then, they are not member variables. Make up your mind.

Would making them const help?

Not if the sketch needs to value the variables defined in other classes.

Your questions are not generic. Time to post some code.

5

Currently they are (regular) member variables of classes

Then you can't put them in PROGMEM.

No, but I can turn them into something else.

Note that I need to use external linkage for these values. They are sketch-specific, i.e., every sketch has to use different values, however their actual use happens inside a library.

Then, they are not member variables. Make up your mind.

They are, but they shouldn't be. Not all decisions make sense.

Changing these members to be actual variables that can be put into flash requires considerable work (mostly political), so I'd rather start the process with the correct solution at hand.

Would making them const help?

Not if the sketch needs to value the variables defined in other classes.

As mentioned above, they don't have to be class members. However, their value needs to be defined in a file that belongs to the sketch while the value is used in a library.

Your questions are not generic.

I found it hard to phrase the question more generic than "How much SRAM do I need for PROGMEM", but I'm working on a minimal code example. Hang on.

Regards,
deltaphi

6

Hello,

here is a minimal example. There is a library (DemoLib) that has a bunch of functions and variables. Depending on what functions are used, only a certain subset of these variables actually has to exist.

#ifndef DEMOLIB__H_
#define DEMOLIB__H_

#include <Arduino.h>

extern uint16_t value1;
extern char value2;

void use_value1();
void use_value2();

#endif

#include <demolib.h>
#include <Arduino.h>

void use_value1() {
	Serial.print(value1);
}

void use_value2() {
	Serial.print(value2);
}

And here are two example sketches using the library:

#include <demolib.h>

uint16_t value1 = 12345;

void setup() {
  Serial.begin(2400);
}

void loop() {
  use_value1();
}

#include <demolib.h>

uint16_t value1 = 12345;
char value2 = 'c';

void setup() {
  Serial.begin(2400);
}

void loop() {
  use_value1();
  use_value2();
}

The goal: Kill value1 and value2 from SRAM in each Sketch. Sketches and library may be modified, under the constraint that the sketch developer does not have to do anything other than define the values once. Using more complex code in the Sketch wrapped in a Macro provided by the library (e.g., #define SET_DEMOLIB_VALUE(name, value) ... or #define SET_VALUE1(value) ...) would be okay.
Minor Edit: Another constraint is that a sketch developer must not have to define values or variables that are not used in the sketch.

Regards,
fantux

PS: Independent of the solution to my problem, I'm still curious about the implementation/SRAM cost of PROGMEM!

7

Well actually if you want to store a large constant array you will save SRAM by using PROGMEM. When you have a const array it is stored in ram unless you specify that it should be stored in flash.
Personally when I want to use a constant variable I just let the pre-processor do the work by using macros instead of const variables.

8

Hello,

Mr_arduino:
When you have a const array it is stored in ram unless you specify that it should be stored in flash.

That is what I am trying to do (except for the "array" part). However, I cannot be sure that PROGMEM'ing value1 and value2 will actually save me anything (as has been pointed out before in this thread), as a pointer the size of value1 (and actually larger than value2) may or may not be involved.

Mr_arduino:
Personally when I want to use a constant variable I just let the pre-processor do the work by using macros instead of const variables.

That would be my first Idea, but with this combination of point of definition and use, using a #define VALUE1 12345 is impossible. demolib.cpp uses the value but does not include DemoSketch*.ino, therefore the preprocessor has no access to VALUE1 when demolib.cpp is compiled.

I have been reading up on this problem for a while now. I saw suggestions that const'ing the variable might lead GCC to optimize the variable away and put the value into Flash. However, this relies on gcc actually performing the optimization and gcc being able to figure out from the current code that this would be a good idea. Furthermore, this optimization was reported to break as soon as external linkage is involved.

Apparently, using "external PROGMEM" is possible - with the open question on associated memory consumption.

Regards,
deltaphi

9

Any help here?
http://www.nongnu.org/avr-libc/user-manual/pgmspace.html

10

Hello,

thank you for this link. It is the best explanation I've seen so far. Unfortunately, it doesn't really tell me anything new. I now know for sure that pointers into Flash are possible. However, I still don't know how the actual PROGMEM'd variable is represented.

To quote from one of the examples in that link:

unsigned char mydata[11][10] PROGMEM =
{ ... ommited for brevity ... }

byte = pgm_read_byte(&(mydata[i][j]));

Is "mydata" a pointer stored in SRAM or a fixed operand in the program code and thus stored fully in Flash until the address computation starts? Sadly, the page doesn't say.

I also took a look at pgmspace.h and found the assembly code that loads data from program memory. The lpm instruction takes the program memory address from a register. However, loading the value into said register is done in C, so I cannot infer how the value is loaded (variable from SRAM or constant from Flash).

Is this just up to one of gcc's optimization algorithms?

Regards,
deltaphi

11

There is no extra pointer. Even though "mydata" is stored in flash, everything works exactly the same from the compiler's viewpoint. "mydata" has an address just like if it were in SRAM. The only difference is, it's not an address in SRAM, it's an address in flash. The compiler doesn't know or care (the compiler thinks there is only one address space). Therefore, pgm_read_* needs to be used, which does understand that a flash address is being given, and executes the appropriate instructions to read from flash.

PS: So there is nothing to stop us from coding as below, and receiving the proverbial Garbage Out.

#include <avr/pgmspace.h>

unsigned char mydata[11][10] PROGMEM =
{
{0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09},
{0x0A,0x0B,0x0C,0x0D,0x0E,0x0F,0x10,0x11,0x12,0x13},
{0x14,0x15,0x16,0x17,0x18,0x19,0x1A,0x1B,0x1C,0x1D},
{0x1E,0x1F,0x20,0x21,0x22,0x23,0x24,0x25,0x26,0x27},
{0x28,0x29,0x2A,0x2B,0x2C,0x2D,0x2E,0x2F,0x30,0x31},
{0x32,0x33,0x34,0x35,0x36,0x37,0x38,0x39,0x3A,0x3B},
{0x3C,0x3D,0x3E,0x3F,0x40,0x41,0x42,0x43,0x44,0x45},
{0x46,0x47,0x48,0x49,0x4A,0x4B,0x4C,0x4D,0x4E,0x4F},
{0x50,0x51,0x52,0x53,0x54,0x55,0x56,0x57,0x58,0x59},
{0x5A,0x5B,0x5C,0x5D,0x5E,0x5F,0x60,0x61,0x62,0x63},
{0x64,0x65,0x66,0x67,0x68,0x69,0x6A,0x6B,0x6C,0x6D}
};

void setup(void)
{
    Serial.begin(9600);

    for (int i = 0; i < 11; i++) {
        for (int j = 0; j < 10; j++) {
            Serial.print(mydata[i][j]);    //wrong! this addresses SRAM, not flash!
            Serial.print('\t');
        }
        Serial.println();
    }
}

void loop(void)
{
}
12

Hello,

thank you for this helpful answer!

To get a better feel for this, I extended my Demo and tried to learn how to use avr-size. This is the code I used in my experiments (modified from my above examples):

#ifndef DEMOLIB__H_
#define DEMOLIB__H_

#include <Arduino.h>

#define USE_PROGMEM
#define USE_CONST

#ifdef USE_CONST
#define MY_CONST const
#else
#define MY_CONST
#endif

#ifdef USE_PROGMEM
#define TYPE_UINT16_T prog_uint16_t PROGMEM
#define TYPE_CHAR prog_char PROGMEM
#else
#define TYPE_UINT16_T uint16_t
#define TYPE_CHAR char
#endif

extern MY_CONST TYPE_UINT16_T value1;
extern MY_CONST TYPE_CHAR value2;


void use_value1();
void use_value2();

#endif

#include <demolib.h>
#include <Arduino.h>

void use_value1() {
	#ifdef USE_PROGMEM
	uint16_t buffer = pgm_read_word(&(value1));
	Serial.print(buffer);
	#else
	Serial.print(value1);
	#endif
}

void use_value2() {
	#ifdef USE_PROGMEM
	char buffer = pgm_read_byte(&(value2));
	Serial.print(buffer);
	#else
	Serial.print(value2);
	#endif
}

#include <demolib.h>

MY_CONST TYPE_UINT16_T value1 = 12345;

void setup() {
  Serial.begin(2400);
}

void loop() {
  use_value1();
}

By fiddling with the #define/#undef to USE_PROGMEM and USE_CONST, I created four test cases, evaluating all combinations of using (or not using) PROGMEM and const.

Results:

Result WITH progmem WITH const

   text    data     bss     dec     hex filename
   2138      16     179    2333     91d DemoSketch1.cpp.elf
   
Result WITH progmem W/O const

   text    data     bss     dec     hex filename
   2138      16     179    2333     91d DemoSketch1.cpp.elf
   
Result W/O progmem WITH const

   text    data     bss     dec     hex filename
   2136      18     179    2333     91d DemoSketch1.cpp.elf
   
Result W/O progmem W/O const

   text    data     bss     dec     hex filename
   2136      18     179    2333     91d DemoSketch1.cpp.elf

First thing to note: YAY! The Size of .data actually goes down by 2 bytes (a uint16_t) when PROGMEM'ing my variable. So this really does solve my problem.

Second thing: Using const has no influence whatsoever on memory usage. Probably due to the external linkage.

In conclusion: Using Progmem does help save SRAM space, even if the variable in question is tiny.

Thank you all very much for your help!

Regards,
deltaphi

13

Just remember, often to do anything useful with that flash-stored variable, you need to place it into a (possibly temporary) SRAM variable. You would then process it or pass it to a function or whatever it is you need to do. So storing single variables in flash may not be very effective in the long run. (And if you're within one or two bytes of running out of memory, you really are cutting things too close anyway.)

PROGMEM comes into its own with arrays of static data, like strings (e.g., "Welcome to My Program v1.0a, compiled on 2014-01-23 by S.Nickity"), or sine tables, gamma correction tables, graphic images, fonts, etc. Assuming these things can be read and processed in small chunks (like one byte at a time), you save an enormous amount of RAM. That's why nearly any example you run across will be storing an array of some sort, not an int.

14