SOLVED: BUG?: PROGMEM works as expected on *global* variables, but NOT *local*

edit, 8 June 2014:
Solved: see replies #7 & 8 below for more info. Thank you Michael Meissner for pointing out the necessity of the static keyword when defining variables that you are using with PROGMEM.

Here are my conclusions: variables must be either global, OR defined w/the static keyword, in order to work with PROGMEM.
-please reply to this issue if you'd like the above info added to the Arduino documentation too. The issue is posted here: Update PROGMEM reference page to mention that variables must be either defined Globally or with the static keyword in order to work w/PROGMEM · Issue #2122 · arduino/Arduino · GitHub

Original post:

I think this is a bug, unless this is part of the intended functionality of PROGMEM. I just lost a couple hours due to locally storing a string constant into PROGMEM. The sketch below shows how locally placing the string into PROGMEM does NOT work, but globally placing the same exact string into PROGMEM DOES work. Is this a bug? Any comments? Thanks!

FYI: for your reference, the PROGMEM documentation is here: PROGMEM - Arduino Reference
and here: avr-libc: <avr/pgmspace.h>: Program Space Utilities

/*
PROGMEM_practice_GLOBAL_VS_LOCAL.ino
By Gabriel Staples
http://electricrcaircraftguy.blogspot.com/
6 June 2014

-The purpose of this sketch is to prove whether or not PROGMEM can be used on LOCAL variables, or if they must be GLOBAL!
--I just lost a couple hrs. to a problem with PROGMEM, when I thought I had a thorough understanding of it, so let's see!
*/

//Global Variables or Constants
//Store a long string into PROGMEM ***GLOBALLY***
PROGMEM char long_str[] = "Hi, I would like to tell you a bit about myself.\n"
                          "My name is Gabriel, and I like to fly RC airplanes.\n"
                          "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                          "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
unsigned int str_len = sizeof(long_str); //string length

void setup()
{
  Serial.begin(115200);
  
  //Store a long string into PROGMEM ***LOCALLY***
  PROGMEM char long_str2[] = "Hi, I would like to tell you a bit about myself.\n"
                            "My name is Gabriel, and I like to fly RC airplanes.\n"
                            "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                            "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
  unsigned int str_len2 = sizeof(long_str2); //string length
  
  
  //print out the two long strings, slowly, one character at a time, in order to make a neat "human-typing" effect
  //FIRST, THE ***GLOBALLY*** DEFINED ONE
  ///////////////////////////////THIS WORKS/////////////////////////////////////////
  for (unsigned int i=0; i<str_len; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str[i]))));
    delay(25);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  //SECOND, THE ***LOCALLY*** DEFINED ONE
  ///////////////////////////////THIS DOES ***NOT*** WORK/////////////////////////////////////////
  for (unsigned int i=0; i<str_len2; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str2[i]))));
    delay(25);
  }
  
} //end of setup()

void loop()
{
  
}

I would imagine it would work if you use the static keyword:

  //Store a long string into PROGMEM ***LOCALLY***
  PROGMEM static char long_str2[] = "Hi, I would like to tell you a bit about myself.\n"
                                    "My name is Gabriel, and I like to fly RC airplanes.\n"
                                    "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                                    "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters

The whole point of PROGMEM is as the ability to store read-only constants in flash memory that you access through a programmatic interface. The AVR has a so-called Harvard architecture , which means pointers to the text area (flash) are in a different address space than pointers to the data area. You have to use different instructions to copy this data to the program.

Even without PROGMEM, you DO NOT want to use an auto array in most cases. Consider the code:

void loop ()
{
  char foo[] = "abcdef";
  // ...
}

Every time the loop function is called, the compiler creates a spot on the stack to hold the array foo. It then must copy the bytes 'a', 'b', 'c', 'd', 'e', 'f', and '\n' into foo. Which means it needs space to hold those bytes, plus the bytes on the stack. If you instead do:

void loop ()
{
  static char foo[] = "abcdef";
  // ...
}

It means the initialization is only done once when the program is linked. For example, if you have the code:

void setup ()
{
  Serial.begin (9600);
}

void loop ()
{
  int x = 0;
  x++;
  Serial.print ("x = ");
  Serial.println (x);
  delay (1000);
}

It will continually print:

x = 1
x = 1
x = 1

If you use static, the initialization is only done once:

void setup ()
{
  Serial.begin (9600);
}

void loop ()
{
  static int x = 0;
  x++;
  Serial.print ("x = ");
  Serial.println (x);
  delay (1000);
}

and it will print out:

x = 1
x = 2
x = 3

Static is like global initialization, except the scope of the variable is only the block it is declared in, so that you can have the same name in different functions, and they represent different values.

It means the initialization is only done once when the program is linked.

It's not done when the program is linked but when the program is started for globals.

Mark

holmes4:

It means the initialization is only done once when the program is linked.

It's not done when the program is linked but when the program is started for globals.

Mark

Yes and no. It puts the initial value of constant data that will be in the SRAM into flash and at program startup it copies the flash to the SRAM. Then it runs the initializers to set up the non-constant data. But I was trying to simplify the answer to explain the difference between static and auto.

Michael,

Thanks for the great explanations. I've been using static for a long time now to do exactly what your demo explained, but it never dawned on me that I should be using it for constant character arrays (strings) too. What about the "const" modifier? Would making it like this work?

  PROGMEM const char long_str2[] = "Hi, I would like to tell you a bit about myself.\n"
                                    "My name is Gabriel, and I like to fly RC airplanes.\n"
                                    "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                                    "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters

or would like this be better? - using static and const?

  PROGMEM static const char long_str2[] = "Hi, I would like to tell you a bit about myself.\n"
                                    "My name is Gabriel, and I like to fly RC airplanes.\n"
                                    "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                                    "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters

panther3001:
Michael,

Thanks for the great explanations. I've been using static for a long time now to do exactly what your demo explained, but it never dawned on me that I should be using it for constant textual arrays (strings) too. What about the "const" modifier?

From what I've read (i.e. I have not looked in the AVR machine dependent portion of the compiler), const is implied by PROGMEM in the 4.3.2 GCC compiler that the Arduino project ships. Due to changes in the way PROGMEM is implemented in GCC 4.5, you will need const if/when Arduino ever updates the compiler (4.9 is the current release BTW). FWIW, the changes were due to infrastructure changes I put into GCC to support multiple address spaces for a different embedded platform (CELL/SPU)>

panther3001:
I think this is a bug, unless this is part of the intended functionality of PROGMEM. I just lost a couple hours due to locally storing a string constant into PROGMEM. The sketch below shows how locally placing the string into PROGMEM does NOT work, but globally placing the same exact string into PROGMEM DOES work. Is this a bug? Any comments? Thanks!

The PSTR() macro will work for local variables, eg.: char *mypstr = PSTR("This string is not copied to ram,");

Thanks all for your comments! I'm marking this as solved, but I am going to request some added documentation to the Arduino PROGMEM page, to have them add a line about it only working on Global or Statically defined Local variables.

Here's my final test code. It shows that variables must be either global, OR defined w/the static keyword, in order to work with PROGMEM.

/*
PROGMEM_practice_IN_DEPTH_TUTORIAL.ino
By Gabriel Staples
http://electricrcaircraftguy.blogspot.com/
6 June 2014

-The purpose of this sketch is to prove whether or not PROGMEM can be used on LOCAL variables, or if they must be GLOBAL!
--I just lost a couple hrs. to a problem with PROGMEM, when I thought I had a thorough understanding of it, so let's see!
*/

//Global Variables or Constants
//1) Store a long string into PROGMEM ***GLOBALLY***
PROGMEM char long_str[] = "Hi, I would like to tell you a bit about myself.\n"
                          "My name is Gabriel, and I like to fly RC airplanes.\n"
                          "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                          "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
unsigned int str_len = sizeof(long_str); //string length

void setup()
{
  Serial.begin(115200);
  
  //2) Store a long string into PROGMEM ***LOCALLY***
  PROGMEM char long_str2[] = "Hi, I would like to tell you a bit about myself.\n"
                            "My name is Gabriel, and I like to fly RC airplanes.\n"
                            "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                            "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
  unsigned int str_len2 = sizeof(long_str2); //string length
  
  //3) Store a long string into PROGMEM ***LOCALLY*** w/the *static* modifier!
  PROGMEM static char long_str3[] = "Hi, I would like to tell you a bit about myself.\n"
                            "My name is Gabriel, and I like to fly RC airplanes.\n"
                            "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                            "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
  unsigned int str_len3 = sizeof(long_str3); //string length
  
  //4) Store a long string into PROGMEM ***LOCALLY*** as a const!
  PROGMEM const char long_str4[] = "Hi, I would like to tell you a bit about myself.\n"
                            "My name is Gabriel, and I like to fly RC airplanes.\n"
                            "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                            "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
  unsigned int str_len4 = sizeof(long_str4); //string length
  
  //5) Store a long string into PROGMEM ***LOCALLY*** as a const AND w/the *static* modifier!
  PROGMEM static const char long_str5[] = "Hi, I would like to tell you a bit about myself.\n"
                            "My name is Gabriel, and I like to fly RC airplanes.\n"
                            "I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!\n"
                            "Random chars: aderowjoeilj feaeowueodhejhnvfer"; //32 random characters
  unsigned int str_len5 = sizeof(long_str5); //string length
  
  //---------------------------------------------------------------------------------------------------------------------
  //print out the two long strings, slowly, one character at a time, in order to make a neat "human-typing" effect
  //---------------------------------------------------------------------------------------------------------------------
  
  //1) FIRST, THE ***GLOBALLY*** DEFINED ONE
  ///////////////////////////////THIS WORKS/////////////////////////////////////////
  Serial.print("str_len = "); Serial.println(str_len);
  for (unsigned int i=0; i<str_len; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str[i]))));
    delay(10);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  
  //2) SECOND, THE ***LOCALLY*** DEFINED ONE
  ///////////////////////////////THIS DOES ***NOT*** WORK/////////////////////////////////////////
  Serial.print("str_len2 = "); Serial.println(str_len2);
  for (unsigned int i=0; i<str_len2; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str2[i]))));
    delay(10);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  
  //3) LOCAL W/STATIC KEYWORD
  ///////////////////////////////THIS WORKS/////////////////////////////////////////
  Serial.print("str_len3 = "); Serial.println(str_len3);
  for (unsigned int i=0; i<str_len3; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str3[i]))));
    delay(10);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  
  //4) LOCAL W/CONST KEYWORD
  ///////////////////////////////THIS DOES ***NOT*** WORK/////////////////////////////////////////
  Serial.print("str_len4 = "); Serial.println(str_len4);
  for (unsigned int i=0; i<str_len4; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str4[i]))));
    delay(10);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  
  //5) LOCAL W/STATIC *AND* CONST KEYWORDS
  ///////////////////////////////THIS WORKS/////////////////////////////////////////
  Serial.print("str_len5 = "); Serial.println(str_len5);
  for (unsigned int i=0; i<str_len5; i++)
  {
    Serial.print(char(pgm_read_byte(&(long_str5[i]))));
    delay(10);
  }
  Serial.println("\n"); //add two line spaces, for good separation
  
} //end of setup()

void loop()
{
  
}

Here is the output in the serial monitor, from the above code:

str_len = 233
Hi, I would like to tell you a bit about myself.
My name is Gabriel, and I like to fly RC airplanes.
I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!
Random chars: aderowjoeilj feaeowueodhejhnvfer

str_len2 = 233
:°‘;&µ¨›À/?ð–¡±?¿º/©/˜/ˆ'‚‘¡±2àˆ™ª»:•Ñ÷Œe? ˆ^“@¸ò”áñ ÈQÜOáñYöß‘Ï‘‘‘ÿ?ï?•Ü\–Œ‘\—ýèñãZÿO ?\–Œ‘\—?à–?s?p\–Œ“\—V–í‘ü‘W— ƒP–í‘ü‘Q—€?€d€ƒ[–œ‘[—\–Œ‘\—˜9ôR–í‘ü‘S—€??}€ƒ•ß’ï’ÿ’““Ï“ß“ìz‹Ò.è‰ù‰‚à€ƒ@0?îX

str_len3 = 233
Hi, I would like to tell you a bit about myself.
My name is Gabriel, and I like to fly RC airplanes.
I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!
Random chars: aderowjoeilj feaeowueodhejhnvfer

str_len4 = 233
a€àha€àxa¡ðàyà?ã?ਗ”ð!P0@@@P@V•G•7•'•€á 08a˜ðè‰ù‰‚ètèŽá?ਗ”ð!P0@@@P@V•G•7•'•ì…ý…0ƒî…ÿ… ƒŽì‰ý‰Ð‚ê‰û‰€?€a€ƒê‰û‰€?ˆ`€ƒê‰û‰€?€h€ƒê‰û‰€??}€ƒß‘Ï‘‘‘ÿ?ï?ß?•ü!?‚?0à ?O(e1 /s0pÉ•ü‘?‚?˜ô/ï?ïÀ‚?èñ…?(/0àÉ•ÜY–œ‘Y—Z–Œ‘

str_len5 = 233
Hi, I would like to tell you a bit about myself.
My name is Gabriel, and I like to fly RC airplanes.
I also really enjoy Arduino microcontroller programming. It is a VERY powerful tool!
Random chars: aderowjoeilj feaeowueodhejhnvfer

Note: the above output didn't display correctly. I think it must contain some null terminators or something, so let me just say that 1, 3, and 5 printed out correctly, but 2 and 4, which were locally defined and NOT defined with the static keyword, did NOT work.

MichaelMeissner:
Even without PROGMEM, you DO NOT want to use an auto array in most cases. Consider the code:

FYI: to future readers who stumble across this, this was the first time someone had ever mentioned to me the idea of an "automatic variable." I didn't know what that meant, so I googled "c auto variable" and got some good hits. Here is one:
Automatic variable - Wikipedia.

Here is another: Difference between static, auto, global and local variable in the context of c and c++ - Stack Overflow

Thanks again Michael!

The same thing applies to PROGMEM members in objects. You must use the static keyword also.

struct Foo{
  static char Data[];
};

char Foo::Data[] PROGMEM = { 'a', 'b', 'c', 0 };

void func() {

  Serial.print( pgm_read_byte( &Foo::Data[ 0 ] ) );

//Or
  Foo f;
  Serial.print( pgm_read_byte( &f.Data[ 0 ] ) );
}

As far as C++ goes, automatic variables are quite powerful. A big difference between C and C++ is the Raii paradigm.

If that link is a bit boggling, consider this, Toggling a pin requires two distinct operations, on & off. Turning on a pin can be a once off at the top of a function. And the off code can be at the end, however if your function has multiple return statements you end up duplicating the off code to ensure every exit point for the function has the 'cleanup code'. The example below has two exit points, therefore requires duplication of the 'off' code:

void func() {

    //Notify user
    digitalWrite( ledPin, HIGH );
    delay( 1000 );
    
    //See if user responds
    if( digitalRead( 14 ) != HIGH ){
      
      digitalWrite( ledPin, LOW );
      return;
    }
    Servo.move( somewhere );
    digitalWrite( ledPin, LOW );
}

To fix this using Raii, we can create a 'Toggle' object:

struct Toggle{
  Toggle( char Pin ) : pin( Pin ) { digitalWrite( pin, HIGH ); }
  ~Toggle() { digitalWrite( pin, LOW ); }
  char pin;
};

void func() {
  
  //Notify user
  const Toggle t( 13 );

  delay( 1000 );
    
   //See if user responds
  if( digitalRead( 10 ) != HIGH ) return;
  servo.write( 256 );
}

As you can see, not only is the functional code cleaner, you now have a guarantee the pin will always be toggled off. If Arduino supported exceptions, then non fatal errors would still have this guarantee met.

This also opens you up to many possibilities:

const Toggle arr[] = { 10, 11, 12, 13 };
Toggle a = 13;
{
  Toggle b = 12;
}

With a slight mod we can toggle a pin x amount of times all from a declaration.

struct Toggle13{
  Toggle13() { digitalWrite( pin, !digitalRead( pin ) ); }
  ~Toggle13() { digitalWrite( pin, !digitalRead( pin ) ); }
  static const char pin = 13;
};
void loop(){
  Toggle13 flash[ 1000 ];
}

You may need to chuck a delay into the constructor to see the togging.

So read up on auto variables, Raii, scope & lifetime. The 'set and forget' nature of Auto variables is a great feature to take advantage of.