printing directly strings from flash (w/o buffer in RAM)

Projects Programming
1

I wonder if there is an easier way to print strings directly from flash without using a buffer or a loop (?)

See my example sketch below.

I define a string in Flash

const char string_0[] PROGMEM = "abcdefghijk";
const char string_1[] PROGMEM = "lmnopqrstu";
// and so on

then I set up a pointer table

const char* const string_table[] PROGMEM = {
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  };

then it's easily possible to print directly from flash (without a buffer) with this "trick" I got from here: array of UTF-8 strings in FLASH (PROGMEM / strcpy_P / ...) - #16 by oqibidipo - Programming Questions - Arduino Forum

// example: flash_println(&string_table[i]);
void flash_println(const char* const* adress)
{
  Serial.println(reinterpret_cast<const __FlashStringHelper *> pgm_read_word(adress));
}

but you have to do that with an indexed access (which is good for large arrays of strings...)

but if I want to access a particular string from flash by its native variable-name
without the use of my pointer table,
how can I do that in a similar "style" (like the function "flash_println()" shown above),
without the use of a buffer or a loop ?

Is that possible ?
(I think it is possible but I am lost between all the helper-macros and typecasts here :slight_smile: )

(the classical way would be to use a loop like it is shown in the sketch below)

the example program:

const char string_0[] PROGMEM = "Check Inputs";
const char string_1[] PROGMEM = "Check Outputs";
const char string_2[] PROGMEM = "Setup/Configuration";
const char string_3[] PROGMEM = "Calibration";
const char string_4[] PROGMEM = "Exit";


// define a table with pointers to the menu strings (stored in flash memory)
const char* const string_table[] PROGMEM = {
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  };




void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  while(!Serial);
  
  // print directly from flash:
  // this works (printing via pointer-table):
  flash_println(&string_table[0]);
  flash_println(&string_table[1]);

  // this doesn't work ():
  //flash_println(&string_0[0]);
  //flash_println(&string_1[0]);

  // this works (using a loop):
  flash_println_classic(&string_0[0]);
  flash_println_classic(&string_1[0]);
  

}



void loop() {
  // put your main code here, to run repeatedly:

}



// print directly from Flash without any buffer:
// from here: https://forum.arduino.cc/index.php?topic=360932.msg2490757#msg2490757
// example: flash_println(&string_table[i]);
void flash_println(const char* const* adress)
{
  Serial.println(reinterpret_cast<const __FlashStringHelper *> pgm_read_word(adress));
}




// the classical way with a loop:
void flash_println_classic(const char* adress) {
  int len = strlen_P(adress);
  char myChar;
  int k;
  
  for (k = 0; k < len; k++)
  {
    myChar =  pgm_read_byte_near(adress + k);
    Serial.print(myChar);
  }

  Serial.println();
 }
2

You probably want the F() macro.

See "Using Flash Memory for string storage" in Arduino Playground - Printf

3

It sounds like you're ready to move from Serial.print() to printf():

#include <stdio.h>
#include <avr/pgmspace.h>

const char string_0[] PROGMEM = "abcdefghijk";
const char string_1[] PROGMEM = "lmnopqrstu";
const char string_2[] PROGMEM = "vwxyzABCDE";
const char string_3[] PROGMEM = "FGHIJKLMNO";
const char string_4[] PROGMEM = "PQRSTUVWXYZ";
const char* const string_table[] PROGMEM = {
  string_0,
  string_1,
  string_2,
  string_3,
  string_4,
  };

int getChar (FILE *fp)
{
    (void)fp;
    while (!(Serial.available()));
    return (Serial.read());
}

int putChar (char c, FILE *fp)
{
    if (c == '\n') {
        putChar ((char) '\r', fp);
    }
    Serial.write (c);
    return 0;
}

void setup()
{
  Serial.begin(9600);
  fdevopen (putChar, getChar);

  for (unsigned int idx = 0; idx < sizeof(string_table)/sizeof(*string_table); idx++) {
    printf("string ptr %d @ %lX points to %x which is string %S \r\n", idx, (unsigned long)&string_table[idx], pgm_read_word(&string_table[idx]), (wchar_t*)pgm_read_word(&string_table[idx]));
  }
}

void loop()
{

}

Output:

string ptr 0 @ 68 points to 9f which is string abcdefghijk
string ptr 1 @ 6A points to 94 which is string lmnopqrstu
string ptr 2 @ 6C points to 89 which is string vwxyzABCDE
string ptr 3 @ 6E points to 7e which is string FGHIJKLMNO
string ptr 4 @ 70 points to 72 which is string PQRSTUVWXYZ

4

Dirk67:
I wonder if there is an easier way to print strings directly from flash without using a buffer or a loop (?)

Sure, just use this Print library: GitHub - krupski/Print: Improved Print functionality for Arduino (including stream support, print_P and related).

Unzip the package and simply replace your current Print.cpp and Print.h with the ones in the package. It fixes a few bugs, adds improvements and adds the ability to print directly from PROGMEM and even EEPROM (EEMEM) by simply using "print_P" and "println_P" (or print_E and println_E for EEPROM).

5

BigBobby:
It sounds like you're ready to move from Serial.print() to printf():

#include <stdio.h>

#include <avr/pgmspace.h>

const char string_0[] PROGMEM = "abcdefghijk";
const char string_1[] PROGMEM = "lmnopqrstu";
const char string_2[] PROGMEM = "vwxyzABCDE";
const char string_3[] PROGMEM = "FGHIJKLMNO";
const char string_4[] PROGMEM = "PQRSTUVWXYZ";
const char* const string_table[] PROGMEM = {
 string_0,
 string_1,
 string_2,
 string_3,
 string_4,
 };

int getChar (FILE *fp)
{
   (void)fp;
   while (!(Serial.available()));
   return (Serial.read());
}

int putChar (char c, FILE *fp)
{
   if (c == '\n') {
       putChar ((char) '\r', fp);
   }
   Serial.write (c);
   return 0;
}

void setup()
{
 Serial.begin(9600);
 fdevopen (putChar, getChar);

for (unsigned int idx = 0; idx < sizeof(string_table)/sizeof(string_table); idx++) {
   printf("string ptr %d @ %lX points to %x which is string %S \r\n", idx, (unsigned long)&string_table[idx], pgm_read_word(&string_table[idx]), (wchar_t)pgm_read_word(&string_table[idx]));
 }
}

void loop()
{

}




Output:

That's a good way to do it (in fact, a lot better than the convoluted way everyone is expect to print).

To make things even easier, use this Stdinout library: GitHub - krupski/Stdinout: Standard input/output/error support for Arduino

Then, writing drivers and "inchar/ outchar" functions for each device is no longer necessary.

Just use "STDIO.open (Serial)" or "STDIO.open (LCD)" or whatever and you have full printf support (less the floating point library - which at least the Arduino devs should provide access to with a Preferences checkbox!!!).

6

sorry, maybe I did not make it clear enough.
my question is just a general programming (learning) question.

it's no problem to use the string_table[] and to have a kind of "indexed" access,
this is shown in my example above (the function flash_println()).
I can see no difference to the indexed access shown by @BigBobby (?)

My question is if I can print out directly by using the variable name (e.g. string_3[]) instead of using the (indexed) string_table[4].

I showed that in my example above in the function "flash_println_classic()" by using a (1-char-) loop.

but my general programming (learning) question is:
can I print out directly from flash by using the variable name (e.g. string_3[])
(instead of using the (indexed access via) string_table[4])
without a buffer or a loop ?

Krupski:
to print directly from PROGMEM and even EEPROM (EEMEM) by simply using "print_P" and "println_P" (or print_E and println_E for EEPROM).

isn't that just another 1-char-loop wrapped into a class or function ? :wink:

b.t.w. I think using printf costs me at least 1000bytes of precious flash (?)

7

That's easy enough. :slight_smile:

const char string_0[] PROGMEM = "Check Inputs";
const char string_1[] PROGMEM = "Check Outputs";
const char string_2[] PROGMEM = "Setup/Configuration";
const char string_3[] PROGMEM = "Calibration";
const char string_4[] PROGMEM = "Exit";

void setup() 
{
  Serial.begin(115200);
  while(!Serial);
  Serial.println ((const __FlashStringHelper *) string_0);
  Serial.println ((const __FlashStringHelper *) string_1);
  Serial.println ((const __FlashStringHelper *) string_2);
  Serial.println ((const __FlashStringHelper *) string_3);
  Serial.println ((const __FlashStringHelper *) string_4);
}

void loop() {}
8

Dirk67:
Is that possible ?

Is it possible that your only programming experiences is on devices using "von Neumann architecture" (like PCs) and you have no experiences on machines with "Harvard architecture"?

Only ""von Neumann" machines mix space for "program instructions" and variable space in the same address space, typically RAM.

In "Harvard architecture" machines program instructions and data are strictly seperated. Either in different parts of RAM, or with Atmegas: Program instructions in flash memory and variables in RAM.

With PROGMEM declaration you put your data into a part of the controller, where only program instructions reside. And if you want to get the data from there into RAM to do something, you have some instructions overhead to get them. PROGMEM data are NOT stored in RAM, they reside in flash memory where normally only program instructions reside.

9

yes,
that's what I'm looking for (I will test that)
(so easy... :blush: )

how to wrap that best into a function by giving the native variable name as a parameter ?
(just for "educational" purposes)

10

Dirk67:
yes,
that's what I'm looking for (I will test that)
(so easy... :blush: )

how to wrap that best into a function by giving the native variable name as a parameter ?
(just for "educational" purposes)

So easy? How about this:

const char pgmstr[] PROGMEM = "Hi there, I am a line in PROGMEM flash memory\n";
const char eemstr[] EEMEM = "Hi there, I am a line in EEPROM memory\n";

void setup (void)
{
    // PSTR must be local (therefore kinda useless).
    const char *pstr[] = {
        PSTR ("Hi there, I am line ONE in PROGMEM memory using PSTR()\n"),
        PSTR ("Hi there, I am line TWO in PROGMEM memory using PSTR()\n"),
    };

    Serial.begin (115200);

    Serial.print_P (pgmstr); // print PROGMEM string directly
    Serial.print_E (eemstr); // print EEPROM string directly
    Serial.print_P (pstr[0]); // print the two PSTR() strings
    Serial.print_P (pstr[1]);
}

void loop (void)
{
    // not used
}

Or this:

#include <Stdinout.h>

void setup (void)
{
    const char *str = "Hi there, I am a line in SRAM memory";
    const int test = 1234;

    Serial.begin (115200);
    STDIO.open (Serial);

    fprintf (stdout, "%s\n", str);
    fprintf (stdout, "The number is %d\n", test);
}

void loop (void)
{
    // not used
}

Neat, huh?

11

Dirk67:
b.t.w. I think using printf costs me at least 1000bytes of precious flash (?)

Results:

#include <Stdinout.h>

void setup (void)
{
    uint32_t mem1;
    uint32_t mem2;
    Serial.begin (115200);

    STDIO.close (); // disable printf, stdin/out, etc...
    mem1 = freeMemory();

    STDIO.open (Serial); // link serial to standard io
    mem2 = freeMemory();

    fprintf (stdout,
        "SRAM usage:\n"
        " without printf: %lu bytes\n"
        " with printf: %lu bytes\n"
        " difference: %lu bytes\n",
        mem1, mem2, (mem1-mem2)
    );
}

void loop (void)
{
    // not used
}

[b]SRAM usage: 
 without printf: 7789 bytes 
 with printf: 7741 bytes 
 difference: 48 bytes[/b]

Yeah... that ram usage is killing me.

If really ARE talking about FLASH (program size), the demo program above uses 3324 bytes of flash (i.e. the "sketch size"). My MEGA2560 is really beginning to sweat.... only has about 259K left.

If you are worried about "precious flash", how about re-writing your bootloader like I did. I reduced it from 8K to 2K simply by removing the garbage in it (not to mention fixing a major bug it has concerning loading EEMEM (eeprom) data into the board. I gained 6144 bytes of "precious" flash memory simply by taking out the trash.

For flash, the "address" variable works as a 16 bit pointer to write two bytes of flash at once, but the people who wrote it never did an "address >> 1" line to correct it for single byte EEPROM writes (and reads for verify).

So, if you try to store an EEMEM string of data, it writes 8 bytes correctly, then SKIPS 8 bytes, then continues writing 8 more, then skips 8, etc.... so in essence with the "factory" bootloader you can only use 1/2 of the EEPROM that the chip has.

I hope EEPROM isn't too precious to you......

12

Dirk67:
how to wrap that best into a function by giving the native variable name as a parameter ?

Krupski's library does that. However if you don't want to install that, you just make a function that does the cast:

const char string_0[] PROGMEM = "Check Inputs";
const char string_1[] PROGMEM = "Check Outputs";
const char string_2[] PROGMEM = "Setup/Configuration";
const char string_3[] PROGMEM = "Calibration";
const char string_4[] PROGMEM = "Exit";

// print_P
size_t print_P (const char *str)
  {
  return Serial.print ((const __FlashStringHelper *) str);
  }

// println_P
size_t println_P (const char *str)
  {
  size_t n = print_P (str);
  return (n + Serial.println());
  }

void setup() 
{
  Serial.begin(115200);
  while(!Serial);
  println_P (string_0);
  println_P (string_1);
  println_P (string_2);
  println_P (string_3);
  println_P (string_4);
}

void loop() {}

Krupski's class is more general. You could use it with Serial1, Serial2 etc.

13

Just noticed this thread; I sort of had the same problem too, and solved it with some macros, see this thread. Not sure if Dirk's problem is the buffer necessary to copy stuff from PROGMEM, but I have no problem using up 100 bytes of dynamic memory to be able to use very many kilobytes of string data from PROGMEM.

My own approach is just a set of simple macros, enabling me to use literal strings all over my sketch without having to declare things const const, or use F() everywhere. Also, there's a macro FS() to pass a string in PROGMEM to C-functions that expect a char *.

I need to be able to use strings in a "direct" manner to control a GSM/GPS, so there are hundreds of AT-commands that I want to use directly in my source code, and not store in vars first, or in some indexed array.

Maybe it was done this way before, but I just did not find it... In that case my apologies :slight_smile:

14

That was the whole point. I wanted to simply do a "STDIO.open (anything)" and not have to think about closing files, freeing memory or anything else.

You can accomplish the same thing by making the "sendchar" and "getchar" functions, then connecting them to the standard streams using "fdevopen()", but if you do it in the wrong order or forget to close a previously opened path, the memory isn't released and you have a memory leak.

I spent a LOT of debugging time with the "freeMemory()" code and lots of open/close/open/close then open something else/close something else etc... to find exactly the proper order to free and allocate the standard paths to be sure things got closed and released and didn't make memory leaks.

For example, if you have stdout and stderr connected to two different paths, you have to close stderr FIRST. Otherwise, if you close stdout first, it also tries to close stderr, but since it's not connected to the same stream, it really doesn't disconnect and it's memory stays allocated.

If you then open different paths, the old ones remain in limbo, still allocated their memory but doing nothing.

I also find the "getStream()" function handy so that I can temporarily use a different stream, then reconnect to where it was without having to keep track of which one was opened last.

They say "necessity is the mother of invention" I disagree. I think LAZINESS is the mother of invention, because I spend a lot of time trying to make things easier for myself. I wonder if there is, in the end, a net gain or a net loss? :slight_smile:

15

Well I've always considered myself extremely lazy. I'm always trying to make things simpler for myself. :slight_smile:

16

[quote author=Nick Gammon link=msg=2702562 date=1460155729][...]you just make a function that does the cast:

const char string_0[] PROGMEM = "Check Inputs";
const char string_1[] PROGMEM = "Check Outputs";
const char string_2[] PROGMEM = "Setup/Configuration";
const char string_3[] PROGMEM = "Calibration";
const char string_4[] PROGMEM = "Exit";

// print_P
size_t print_P (const char *str)
  {
  return Serial.print ((const __FlashStringHelper *) str);
  }

// println_P
size_t println_P (const char *str)
  {
  size_t n = print_P (str);
  return (n + Serial.println());
  }

void setup() 
{
  Serial.begin(115200);
  while(!Serial);
  println_P (string_0);
  println_P (string_1);
  println_P (string_2);
  println_P (string_3);
  println_P (string_4);
}

void loop() {}

[/quote]

Thanks again @Nick Gammon
that's exactly what I was looking for.

17

Is there any functional difference between these two:

size_t Print::print (const __FlashStringHelper *ifsh)
{
    PGM_P p = reinterpret_cast<PGM_P> (ifsh);
    size_t n = 0;
    unsigned char c;

    while ((c = pgm_read_byte (p + n++))) {
        write (c);
    }

    return n;
}

and...

size_t Print::print_P (const char *str)
{
    size_t n = 0;
    char c;
    while ((c = pgm_read_byte (str + n++))) {
        write (c);
    }

    return n;
}

?

If so, could you please explain it to me. Thanks.

18

Apart from the fact that the functions take different types as parameters?

19

I'm not familiar with (const __FlashStringHelper *)

Is there a reference for this as opposed to descriptions of how to use it?
Does it really print directly from flash without buffering (serial output buffer doesn't count)?

My current print from flash takes a PGM_P and uses 1 byte of RAM to print the string.
Will this (const __FlashStringHelper *) run faster and save using that byte?

20

With the (const __FlashStringHelper *) it works by the same way. It reads a byte to RAM and writes to the output.