store and retrieve array of chars in flash memory

renanlopesmister · May 2, 2018, 3:22am

Hello everyone!
First of all, I'm sorry for my English!

I'm trying:

storing strings (char arrays) in flash memory
read and store these strings in the same buffer always (another array char);
concatenate this buffer and another of these strings;
print this buffer, after all this process, on the serial monitor.

The storage of strings in the buffer and the concatenations are done through a lib I wrote.

This is my code:

#include <MyString.h>
MyString string;



const char STR_0[] PROGMEM = "string0_";
const char STR_1[] PROGMEM = "string1_";
const char STR_2[] PROGMEM = "string2_";
const char STR_3[] PROGMEM = "string3_";
const char STR_4[] PROGMEM = "string4_";
const char STR_5[] PROGMEM = "string5_";

const char* const STR[] PROGMEM = {
 STR_0,
 STR_1,
 STR_2,
 STR_3,
 STR_4,
 STR_5
};

char buff[17];



void function() {
 /* 
 string.put_P(buff, STR[0]);
 string.catenate_P(buff, STR[1]);
 Serial.println(buff);
 
 string.put_P(buff, STR[1]);
 string.catenate_P(buff, STR[2]);
 Serial.println(buff);
 
 string.put_P(buff, STR[2]);
 string.catenate_P(buff, STR[3]);
 Serial.println(buff);
 
 string.put_P(buff, STR[3]);
 string.catenate_P(buff, STR[4]);
 Serial.println(buff);
 
 string.put_P(buff, STR[4]);
 string.catenate_P(buff, STR[5]);
 Serial.println(buff);
 
 Serial.println();
 */
 
 byte lastSTR = (sizeof(STR) / sizeof(char *)) - 1;
 
 for(byte i = 0; i < lastSTR; i++) {
 string.put_P(buff, STR[i]);
 string.catenate_P(buff, STR[i + 1]);
 Serial.println(buff);
 }
 
 Serial.println();
}



void setup() {
 Serial.begin(9600);
}


void loop() {
 function();
}

The lines that are commented on seem to work fine. Just below, I tried to optimize the code with a for loop, but the result on the serial monitor is 2 or 3 random characters!
The functions of my lib I am using are these:

void MyString :: put_P(char *buffer, const char *str) {
 byte i; // counter;
 
 for(i = 0; pgm_read_byte(str + i); i++) buffer[i] = pgm_read_byte(str + i);
 
 buffer[i] = '\0';
} 


void MyString :: catenate_P(char *str0, const char *str1) {
 byte i, j; // counters;
 
 for(i = 0; str0[i]; i++); // identifies the end of str0;
 
 for(j = 0; pgm_read_byte(str1 + j); j++) str0[i + j] = pgm_read_byte(str1 + j);
 
 str0[i + j] = '\0';
}

Why does the commented code work and the for loop does not?

Thank you in advance for your attention

system · May 2, 2018, 8:03am

Why does the commented code work and the for loop does not?

You REALLY need to explain this statement. The commented out code calls the function with the for loop. The call can't POSSIBLY work of the function does something stupid.

renanlopesmister · May 2, 2018, 12:00pm

I will show the result of each situation. Below the code that seems to be working the way I want it:

// my lib
#include <MyString.h>
MyString string;


// strings I want to save in flash
const char STR_0[] PROGMEM = "string0_";
const char STR_1[] PROGMEM = "string1_";
const char STR_2[] PROGMEM = "string2_";
const char STR_3[] PROGMEM = "string3_";
const char STR_4[] PROGMEM = "string4_";
const char STR_5[] PROGMEM = "string5_";

const char* const STR[] PROGMEM = {
STR_0,
STR_1,
STR_2,
STR_3,
STR_4,
STR_5
};

// buffer will save 2 concatenated strings
char buff[17];



void function() {
string.put_P(buff, STR[0]);
string.catenate_P(buff, STR[1]);
Serial.println(buff);

string.put_P(buff, STR[1]);
string.catenate_P(buff, STR[2]);
Serial.println(buff);

string.put_P(buff, STR[2]);
string.catenate_P(buff, STR[3]);
Serial.println(buff);

string.put_P(buff, STR[3]);
string.catenate_P(buff, STR[4]);
Serial.println(buff);

string.put_P(buff, STR[4]);
string.catenate_P(buff, STR[5]);
Serial.println(buff);

Serial.println();
}



void setup() {
Serial.begin(9600);
}



void loop() {
function();
}

[Album] imgur.com

Now, the optimization with the for loop that did not present the expected result:

// my lib
#include <MyString.h>
MyString string;


// strings I want to save in flash
const char STR_0[] PROGMEM = "string0_";
const char STR_1[] PROGMEM = "string1_";
const char STR_2[] PROGMEM = "string2_";
const char STR_3[] PROGMEM = "string3_";
const char STR_4[] PROGMEM = "string4_";
const char STR_5[] PROGMEM = "string5_";

const char* const STR[] PROGMEM = {
STR_0,
STR_1,
STR_2,
STR_3,
STR_4,
STR_5
};

// buffer will save 2 concatenated strings
char buff[17];



void function() {
byte lastSTR = (sizeof(STR) / sizeof(char *)) - 1;

for(byte i = 0; i < lastSTR; i++) {
string.put_P(buff, STR[i]);
string.catenate_P(buff, STR[i + 1]);
Serial.println(buff);
}

Serial.println();
}



void setup() {
Serial.begin(9600);
}



void loop() {
function();
}

[Album] imgur.com

Why does the first code seem to work correctly?

system · May 2, 2018, 12:24pm

Imgur: The magic of the Internet

Why would you post tiny pictures of text?

renanlopesmister · May 2, 2018, 12:39pm

I simulated these codes in proteus and printed the screen in each case. It was not very good to host these impressions on this site. Should I edit my post?

Danois90 · May 2, 2018, 12:40pm

The whole idea of using PROGMEM (flash) for variables is to avoid them from taking up space in RAM. What you are doing is to have the variables stored both in flash and in RAM which is contra-productive. If you need to manipulate strings, store them in RAM

renanlopesmister · May 2, 2018, 12:54pm

Danois90:
If you need to manipulate strings, store them in RAM

What I intend is to occupy the RAM only with the necessary string for a certain stage of the program. I thought it would be interesting to store all the strings in the flash and refresh "buff [17]" when it would be necessary to read a new string. I'm having a hard time doing this smarter.

Koepel · May 2, 2018, 12:57pm

Who wrote MyString.h and MyString.cpp ? and may I see that whole file please ?
When we want to try your sketch, we need the latest version of all the files that you use.

The keyword "String" is used for the String object: String() - Arduino Reference.

When you are using normal character arrays, then all of those functions already exist and they can be used in a normal way.
All those functions are here: avr-libc: <string.h>: Strings

The special PROGMEM functions are here: avr-libc: <avr/pgmspace.h>: Program Space Utilities

Are you sure that you have calculated the lastSTR properly ? Why strip the last element with "- 1" ?

You can start with something that works:

// about 6 strings of 8 bytes each fits in an array of 80 bytes.
char buff[80];

void function() 
{
  buff[0] = '\0';   // make empty string to start with
  int n = sizeof(STR) / sizeof(char *);

  for(int i = 0; i < n; i++) 
  {
    strcat_P( buff, (const char *) pgm_read_word(&(STR[i])));
    Serial.println(buff);
  }
  Serial.println();
}

Did you read the PROGMEM - Arduino Reference and Gammon Forum : Electronics : Microprocessors : Putting constant data into program memory (PROGMEM).

When you want to concatenate two strings, the source can be in flash (PROGMEM), but the destination must be RAM. That means the buffer has to be large enough to hold the result of all the concatenation.

dev_1 · May 2, 2018, 1:36pm

Just print the pieces:

// strings I want to save in flash
const char STR_0[] PROGMEM = "string0_";
const char STR_1[] PROGMEM = "string1_";
const char STR_2[] PROGMEM = "string2_";
const char STR_3[] PROGMEM = "string3_";
const char STR_4[] PROGMEM = "string4_";
const char STR_5[] PROGMEM = "string5_";

const char* const STR[] PROGMEM =
  {
    STR_0,
    STR_1,
    STR_2,
    STR_3,
    STR_4,
    STR_5
  };

const size_t MAX_STRINGS = sizeof(STR) / sizeof(STR[0]);

// Handy macro for casting the "const char *" to the correct type for printing
#define CF(s) ((const __FlashStringHelper *)s)

void function()
{
  uint8_t i=0;

  while (i<MAX_STRINGS) {
    Serial.print( CF( STR[i] ) );
    i++;

    if (i<MAX_STRINGS)
      Serial.print( CF( STR[i] ) );
    i++;

    Serial.println();
  }

  Serial.println();
}



void setup() {
  Serial.begin(9600);
  function();
}


void loop() {
}

There's no reason to make a big RAM copy of the FLASH strings.

renanlopesmister · May 2, 2018, 1:55pm

Who wrote MyString.h and MyString.cpp ?

I wrote this little lib, because when I started writing this scketch, I was not sure if the strlib commands were already integrated into the Arduino library. I have already taken the initiative to write my own commands. Also, when I created another sketch that involves many LCD printing functions, for some reason, which I did not debug, the sketch showed some bugs, and with my lib everything went well apparently. So I kept using my lib.

When we want to try your sketch, we need the latest version of all the files that you use.

MyString.h

#ifndef MyString_h
#define MyString_h
#include "Arduino.h"



class MyString {
	public:
		MyString();
		
		void
			put(char *buffer, const char *str),
			catenate(char *str0, const char *str1),
			catenate(char *str, const char chr),
			catenate(const char chr, char *str),
			put_P(char *buffer, const char *str),
			catenate_P(char *str0, const char *str1);
		
		private:
		
};



#endif

MyString.cpp

#include "Arduino.h"
#include "MyString.h"



MyString :: MyString() {

}



void MyString :: put(char *buffer, const char *str) {
	byte i; // counter;
	
	for(i = 0; str[i]; i++) buffer[i] = str[i];
	
	buffer[i] = '\0';
} 



void MyString :: catenate(char *str0, const char *str1) {
	byte i, j; // counters;
	
	for(i = 0; str0[i]; i++);
	
	for(j = 0; str1[j]; j++) str0[i + j] = str1[j];
	
	str0[i + j] = '\0';
} 



void MyString :: catenate(char *str, const char chr) {
	byte i; // counter;
	
	for (i = 0; str[i]; i++);
	
	str[i] = chr;
	str[i + 1] = '\0';
} 



void MyString :: catenate(const char chr, char *str) {
	byte i; // counter;
	
	for(i = 0; str[i]; i++);
	i++;
	
	for( ; i; i--) str[i] = str[i - 1];
	
	str[0] = chr;
}



void MyString :: put_P(char *buffer, const char *str) {
	byte i; // counter;
	
	for(i = 0; pgm_read_byte(str + i); i++) buffer[i] = pgm_read_byte(str + i);
	
	buffer[i] = '\0';
} 



void MyString :: catenate_P(char *str0, const char *str1) {
	byte i, j; // counters;
	
	for(i = 0; str0[i]; i++); // identifies the end of str0;
	
	for(j = 0; pgm_read_byte(str1 + j); j++)
		str0[i + j] = pgm_read_byte(str1 + j);
	
	str0[i + j] = '\0';
}

Are you sure that you have calculated the lastSTR properly ? Why strip the last element with "- 1" ?

The last concatenation of the for loop should be between the penultimate and last element of the STR []

You can start with something that works:

I'm beginning to understand the correct way to use the pgm_read commands. I am interpreting your suggested code as I reread the references.

renanlopesmister · May 2, 2018, 3:05pm

-dev:
Just print the pieces:

There's no reason to make a big RAM copy of the FLASH strings.

But if I want to manipulate some of these strings in some way, I need to copy them to RAM, right?

sterretje · May 2, 2018, 4:59pm

renanlopesmister:
But if I want to manipulate some of these strings in some way, I need to copy them to RAM, right?

Yes. But what would you want to change? Concatenation is no reason as wherever (lcd, serial, ethernet, file) you send a concatenation of two strings, you can send them individually.

dev_1 · May 2, 2018, 5:35pm

But if I want to manipulate some of these strings in some way, I need to copy them to RAM, right?

Maybe. Filtering and parsing does not require copying. What are you trying to do?

Danois90 · May 2, 2018, 5:41pm

The smart sollution would be not to load flash into RAM in order to manipulate it. The smart sollution would be to implement a function which could manipulate the flash variables on a byte-by-byte basis without buffering

sterretje · May 2, 2018, 5:52pm

Danois90:
The smart sollution would be not to load flash into RAM in order to manipulate it. The smart sollution would be to implement a function which could manipulate the flash variables on a byte-by-byte basis without buffering

Which is not possible on AVR based Arduinos without modifying the bootloader or other tricks. And flash has an estimated life cycle if 10000 writes.

renanlopesmister · May 2, 2018, 6:28pm

With all these suggestions and questions, I conclude that I need to rethink the logic of my scketch. But I was intrigued by the fact that with the first code from post # 2 I got the result I wanted and with the second code I could not, and they seem to do the same thing exactly

Danois90 · May 2, 2018, 6:44pm

sterretje:
Which is not possible on AVR based Arduinos without modifying the bootloader or other tricks. And flash has an estimated life cycle if 10000 writes.

You may have misunderstood what I meant. I was not suggesting to modify the flash, but to load the strings from flash byte-by-byte and modify the individual bytes in RAM before using them. If OP wanted to be able to combine different strings in different order, he could:

void print_PSTR(byte index)
{
  for (i = 0; i < strlen; i++)
  {
    byte pstr_byte = load_PSTR_BYTE(STR[index][i]);
    //Modify the byte ont-the-fly
    print(pstr_byte);
  }
}

void combine_PSTR(byte index1, byte index2)
{
  print_PSTR(index1);
  print_PSTR(index2);
}

And so on..

dev_1 · May 2, 2018, 7:32pm

sterretje:
"The smart solution would be to implement a function which could manipulate the flash variables on a byte-by-byte basis without buffering

Which is not possible on AVR based Arduinos without modifying the bootloader or other tricks.

He means reading a byte of FLASH and deciding whether it should be passed on to its final destination or not, or with modification. (I see Danois90 has clarified.) For example:

// strings I want to save in flash
const char STR_0[] PROGMEM = "string0_";
const char STR_1[] PROGMEM = "string1_";
const char STR_2[] PROGMEM = "string2_";
const char STR_3[] PROGMEM = "string3_";
const char STR_4[] PROGMEM = "string4_";
const char STR_5[] PROGMEM = "string5_";

const char* const STR[] PROGMEM =
 {
   STR_0,
   STR_1,
   STR_2,
   STR_3,
   STR_4,
   STR_5
 };

const size_t MAX_STRINGS = sizeof(STR) / sizeof(STR[0]);

// Handy macro for casting the "const char *" to the correct type for printing
#define CF(s) ((const __FlashStringHelper *)s)

void function()
{
 uint8_t i=0;

 while (i<MAX_STRINGS) {
   printUppercaseFLASHstr( STR[i] );
   i++;

   if (i<MAX_STRINGS)
     printUppercaseFLASHstr( STR[i] );
   i++;

   Serial.println();
 }

 Serial.println();
}

void printUppercaseFLASHstr( const char *flashPtr )
{
 for (;;) {
   char c = pgm_read_byte( flashPtr++ );
   if (c == '\0')
     break;

   if (('a' <= c) and (c <= 'z'))
     c -= 'a' - 'A'; // offset is 32, so subtracting 32 from 'd' (80) is a 'D' (68)
   Serial.write( c );
 }
}


void setup() {
 Serial.begin(9600);
 function();
}


void loop() {
}

This prints the uppercase versions of all those strings without ever copying them to a RAM buffer. The output:

STRING0_STRING1_
STRING2_STRING3_
STRING4_STRING5_

Actually, I suspect the sketch performs everything in registers, so it may not use any RAM.

renanlopesmister · May 2, 2018, 10:08pm

Well, I need to redo all my print routines so that these routines, instead of receiving an array of RAM characters as a parameter, get a byte-byte reading of the strings in flash memory, for example, and print as desired. This will be quite laborious, but my scketch is not working anyway ...

But why can I pass the strings, which are in flash memory, to a buffer one way and not another?

sterretje · May 3, 2018, 4:47am

renanlopesmister:
Well, I need to redo all my print routines so that these routines, instead of receiving an array of RAM characters as a parameter, get a byte-byte reading of the strings in flash memory, for example, and print as desired. This will be quite laborious, but my scketch is not working anyway ...

Why byte-by-byte? It's still not clear to me what you want to achieve. For simple concatenation of two strings that are stored in flash, see reply #7.

In the below, I used strcpy_P (I forgot about strcat) which will give the same result; the strings are stored in flash but the pointer array is however stored in ram.

const char hello[] PROGMEM = "Hello";
const char space[] PROGMEM = " ";
const char world[] PROGMEM = "world";

const char *texts[] =
{
  hello,
  space,
  world,
};

void foo(char *b, const char *t[], int numElements)
{
  for (int cnt = 0; cnt < numElements; cnt++)
  {
    strcpy_P(&b[strlen(b)], t[cnt]);
  }
}

void setup()
{
  Serial.begin(57600);
  char buffer[64];
  buffer[0] = '\0';
  strcpy_P(buffer, texts[0]);
  Serial.print("'"); Serial.print(buffer); Serial.println("'");
  strcpy_P(&buffer[strlen(buffer)], texts[1]);
  Serial.print("'"); Serial.print(buffer); Serial.println("'");
  strcpy_P(&buffer[strlen(buffer)], texts[2]);
  Serial.print("'"); Serial.print(buffer); Serial.println("'");

  buffer[0] = '\0';
  foo(buffer, texts, sizeof(texts) / sizeof(texts[0]));
  Serial.print("'"); Serial.print(buffer); Serial.println("'");
}

void loop()
{
}

renanlopesmister:
But why can I pass the strings, which are in flash memory, to a buffer one way and not another?

Do you mean, why can I read but not write?

Topic		Replies	Views
Question about PROGMEM (strcpy_P usage) Programming	74	850	July 22, 2025
Working with array of strings, PROGMEM question Programming	15	14997	May 5, 2021
F() macro and PROGMEM Programming	60	74350	May 5, 2021
PROGMEM const not working Programming	16	5900	May 5, 2021
Print string characters from FLASH Programming	10	1215	May 5, 2021

store and retrieve array of chars in flash memory

Related topics