F() macro and sprintf

WPD64 · August 11, 2021, 10:13am

Hi everybody

Edit:
Platform is Arduino MKR Wifi 1010

It was my understanding that you cannot pass __FlashStringHelper type arguments to the variable-length argument list of sprintf().

And it was my understanding that one should use sprintf_P() and the (capitalized) %S format-identifier for Flash-located strings (const char*).

My experimenting and the code below however are telling a different story.
What am I not understanding here??

  char buf[256];
  snprintf(buf,255,"snprintf-s: %s",F("PROGMEM-STRING"));
  Serial.println(buf);
  
  snprintf(buf,255,"snprintf-S: %S",F("PROGMEM-STRING"));
  Serial.println(buf);

  snprintf_P(buf,255,"snprintf_P-s: %s",F("PROGMEM-STRING"));
  Serial.println(buf);
  
  snprintf_P(buf,255,"snprintf_P-S: %S",F("PROGMEM-STRING"));
  Serial.println(buf);

Output:
snprintf-s: PROGMEM-STRING
snprintf-S: S
snprintf_P-s: PROGMEM-STRING
snprintf_P-S: S

Whandall · August 11, 2021, 11:26am

warning: format '%S' expects argument of type 'wchar_t*', but argument 4 has type 'const char*'

so "%S" is not related to PROGMEM in any way.

snprintf_P expects the format string in PROGMEM.

The optimizer will probably make some strange combinations work.

Koepel · August 11, 2021, 12:04pm

No one could show a small demonstration sketch that I asked for here: https://forum.arduino.cc/t/resolved-without-hacks-how-i-used-f-flash-memory-strings-in-snprintf-etc-for-arduino-nano/891927/14
And I could not make it work myself.

So if no one can give some prove, then Whandall is right and the others are wrong and it is wrong information or fake or bad documentation.

The Arduino MKR Wifi 1010 does not need PROGMEM. I think it is enough if you make a string a 'const'.

Juraj · August 11, 2021, 12:12pm

so it is

johnwasser · August 11, 2021, 12:29pm

The %S/PROGMEM is a feature of avr libc so it only pertains to avr processors.

https://www.nongnu.org/avr-libc/user-manual/group__avr__stdio.html#gaa3b98c0d17b35642c0f3e4649092b9f1

S Similar to the s format, except the pointer is expected to point to a program-memory (ROM) string instead of a RAM string.

Whandall · August 11, 2021, 12:32pm

That message was generated while compiling for a Nano.

johnwasser · August 11, 2021, 1:02pm

I just looked at the avr libc sources for vfprintf.c and it has support for %S->PROGMEM string.

http://cvs.savannah.nongnu.org/viewvc/avr-libc/avr-libc/libc/stdio/vfprintf.c?view=log

It looks like %S->wchar got added to the gcc compiler and conflicts with the avr-libc use of %S->PROGMEM string.

WPD64 · August 11, 2021, 1:04pm

The Arduino MKR Wifi 1010 does not need PROGMEM. I think it is enough if you make a string a 'const'.

So you are saying, all const char *ptr are pointing into program memory?

WPD64 · August 11, 2021, 1:12pm

The optimizer will probably make some strange combinations work.

Two questions:
Is it possible to prevent the optimizer from doing its magic?

How will I ever know if am using my pointers correctly, if even const char* blobs are residing in program memory (as Koepel claimed)?

johnwasser · August 11, 2021, 1:51pm

Interesting... Looks like the 'wchar' warning only happens for snprintf(), not snprintf_P().

Also, the warnings do not prevent the compiler from generating the right code. When the pointer's address space matches the format string, the string is displayed correctly. When the pointer's address space does not match the format string, the string is not displayed correctly.

char buf[256];

void setup()
{
  Serial.begin(115200);
  delay(200);

  Serial.println("Valid pointers:");

  snprintf(buf, 255, "snprintf-s: %s", "NORMAL-STRING");
  Serial.println(buf);

  snprintf(buf, 255, "snprintf-S: %S", PSTR("PSTR-STRING"));
  Serial.println(buf);

  snprintf(buf, 255, "snprintf-S: %S", (const char *) F("F_MACRO-STRING"));
  Serial.println(buf);

  snprintf_P(buf, 255, PSTR("snprintf_P-s: %s"), "NORMAL-STRING");
  Serial.println(buf);

  snprintf_P(buf, 255, PSTR("snprintf_P-S: %S"), PSTR("PSTR-STRING"));
  Serial.println(buf);

  snprintf_P(buf, 255, (const char *) F("snprintf_P-S: %S"), (const char *) F("F_MACRO-STRING"));
  Serial.println(buf);

  Serial.println("Invalid pointers:");

  snprintf(buf, 255, "snprintf-s: %s", PSTR("PSTR-STRING"));
  Serial.println(buf);

  snprintf(buf, 255, "snprintf-s: %s", (const char *) F("F_MACRO-STRING"));
  Serial.println(buf);

  snprintf(buf, 255, "snprintf-S: %S", "NORMAL-STRING");
  Serial.println(buf);
  

  snprintf_P(buf, 255, PSTR("snprintf_P-s: %s"), PSTR("PSTR-STRING"));
  Serial.println(buf);
  
  snprintf_P(buf, 255, PSTR("snprintf_P-s: %s"), (const char *) F("F_MACRO-STRING"));
  Serial.println(buf);

  snprintf_P(buf, 255, PSTR("snprintf_P-S: %S"), "NORMAL-STRING");
  Serial.println(buf);
}

void loop() {}

Run on an Arduino UNO:

Valid pointers:
snprintf-s: NORMAL-STRING
snprintf-S: PSTR-STRING
snprintf-S: F_MACRO-STRING
snprintf_P-s: NORMAL-STRING
snprintf_P-S: PSTR-STRING
snprintf_P-S: F_MACRO-STRING
Invalid pointers:
snprintf-s: 
snprintf-s: ⸮⸮⸮
snprintf-S: F_MACRO-STRING
snprintf_P-s: 
snprintf_P-s: 
snprintf_P-S: F_MACRO-STRING

NOTE: The two appearances of "F_MACRO-STRING" under "Invalid pointers:" were RAM pointers to "NORMAL-STRING". Looks like the address of "NORMAL-STRING" in RAM just happened to match the address of "F_MACRO-STRING" in PROGMEM.

The compiler warnings are:

/Users/john/Documents/Arduino/sketch_aug11a/sketch_aug11a.ino: In function 'void setup()':
/Users/john/Documents/Arduino/sketch_aug11a/sketch_aug11a.ino:13:59: warning: format '%S' expects argument of type 'wchar_t*', but argument 4 has type 'const char*' [-Wformat=]
   snprintf(buf, 255, "snprintf-S: %S", PSTR("PSTR-STRING"));
                                                           ^
/Users/john/Documents/Arduino/sketch_aug11a/sketch_aug11a.ino:16:74: warning: format '%S' expects argument of type 'wchar_t*', but argument 4 has type 'const char*' [-Wformat=]
   snprintf(buf, 255, "snprintf-S: %S", (const char *) F("F_MACRO-STRING"));
                                                                          ^
/Users/john/Documents/Arduino/sketch_aug11a/sketch_aug11a.ino:37:55: warning: format '%S' expects argument of type 'wchar_t*', but argument 4 has type 'const char*' [-Wformat=]
   snprintf(buf, 255, "snprintf-S: %S", "NORMAL-STRING");
                                                       ^
Sketch uses 3980 bytes (12%) of program storage space. Maximum is 32256 bytes.
Global variables use 522 bytes (25%) of dynamic memory, leaving 1526 bytes for local variables. Maximum is 2048 bytes.

Perhaps the "-Wformat" argument to the compiler could be used to suppress the spurious warnings.

Koepel · August 11, 2021, 1:54pm

@WPD64, not only the 'const' is in Flash memory, but you should also not use the "%S", just the normal "%s".

The AVR family microcontrollers, such as the Arduino Uno, have a simple and short instruction set. They can not execute code from RAM and they can not have a pointer to data in Flash memory. That is why those special functions, such as 'pgm_read_byte()' are needed to be able to get that data.

The SAMD family processors are more advanced processors, they can use data in Flash.

Make a small test sketch, and try with 'const' and 'PROGMEM' and see what the compiler says about dynamic memory (RAM) usage. Turn on 'verbose' output on 'Compilation' in the preferences.

const char pText[] PROGMEM = 
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789"
  "0123456789012345678901234567890123456789012345678901234567890123456789";

void setup()
{
  Serial.begin( 9600);
  Serial.println( pText);
}

void loop() {}

Arduino Uno:
char pText[] : 888 bytes
const char pText[] : 888
const char pText[] PROGMEM : 188 (data is in Flash memory)

Arduino MKR Wifi 1010:
char pText[] : 2908
const char pText[] : 2204 (data is in Flash memory)
const char pText[] PROGMEM : 2204

The 'PROGMEM' does not give an error with the MKR board, because they made the source code compatible, but it is not needed.

Koepel · August 11, 2021, 2:09pm

@johnwasser Thank you !
I think I know what I did wrong, when one of the invalid ones is in the sketch, that messes up the ram so bad, that the valid ones will crash.

You missed a few. This one is valid as well:

const char format[] PROGMEM = "snprintf_P-PROGMEM-S-PROGMEM: %S";
const char hello[] PROGMEM = "Hello World";
...
  snprintf_P(buf, 255, format, hello);

@WPD64 this is for the Arduino Uno, not for your board.

WPD64 · August 11, 2021, 2:46pm

I did and can confirm that for const char* and the PROGMEM option will consume FLASH instead of RAM.

But I do not understand how these incompatible pointers are resolved at runtime.
Unless the SAMD-platform employs virtual memory or some other transparent memory-management.

WPD64 · August 11, 2021, 2:52pm

Just to confirm that I understand correctly:

On the SAMD-platform:

There's no need at all for PROGMEN or F() to reduce RAM consumption.
All const char* will reside in FLASH.
snprintf and the like will be fine with (transparently) FLASH-based strings.

J-M-L · August 11, 2021, 3:01pm

That's up to the compiler to decide where to store the data. the C++ specification does not say so

MicroBahner · August 11, 2021, 3:25pm

The reason for this behavior ist the complete different architectures of AVR and SAMD processors. The AVR processors have a strict harvard architecture. That means, that programm memory and data memory are completly apart from another and they both have their own adress space, starting from 0. Which type of memory is addressed does not depend from the adress but it depends only on the machine code used. There are only very few machine codes to fetch data from program memory. To create the correct machine code, the compiler must not only know an address, it must also know wether this address is located im program or data memory.

In contrast the SAMD (=arm) processors only have one contigous address space, and part of it is flash and another part is ram. Every machine code can access ram or flash, it is distinguished only by the address ( of course, code that writes to memory will fail in the address range of flash memory ). Therefore the compiler can address (constant) strings in flash in the same manner as in ram.

WPD64 · August 11, 2021, 3:48pm

Duh - C/C++ has no concept of these things

johnwasser · August 11, 2021, 3:48pm

They are incompatible on the avr processors because RAM, PROGMEM, and EEPROM have separate address spaces. On a processor with only one address space, all pointers are compatible.

WPD64 · August 11, 2021, 3:58pm

Got it.
But why is this distinction even made and why is the concrete location of objects left unclear to the developer while at the same time imposing a strict limit on their respective sizes??

WPD64 · August 11, 2021, 3:58pm

Got it.

Topic		Replies	Views
<RESOLVED without hacks > How I used F() Flash Memory Strings in snprintf, etc for Arduino Nano Programming	21	2797	December 9, 2021
Question about PROGMEM (strcpy_P usage) Programming	74	620	July 22, 2025
F() macro and PROGMEM Programming	60	74204	May 5, 2021
What does the F() do exactly? Programming	74	144936	May 5, 2021
Current situation with PROGMEM etc. Programming	34	1773	May 6, 2021

F() macro and sprintf

Related topics