What does the F() do exactly?

I just accept it as a fact of the compiler because the end result saved me 1620 Bytes of data area at the cost of 704 bytes of Flash memory.

I find it hard to believe that using F() cost you 704 bytes of flash. I submit this example:

size_t freeRam ()
{
  return RAMEND - size_t (__malloc_heap_start);
} // end of freeRam


void setup ()
{
  Serial.begin (115200);
  Serial.println ();
  Serial.print ("Free RAM is ");
  Serial.println ((int)freeRam);
  Serial.println (
      "  \"Jabberwocky\"\n"
        "\n"
        "'Twas brillig, and the slithy toves\n"
        "Did gyre and gimble in the wabe;\n"
        "All mimsy were the borogoves,\n"
        "And the mome raths outgrabe.\n"
        "\n"
        "\"Beware the Jabberwock, my son!\n"
        "The jaws that bite, the claws that catch!\n"
        "Beware the Jubjub bird, and shun\n"
        "The frumious Bandersnatch!\"\n"
        "\n"
        "He took his vorpal sword in hand:\n"
        "Long time the manxome foe he sought—\n"
        "So rested he by the Tumtum tree,\n"
        "And stood awhile in thought.\n"
        "\n"
        "And as in uffish thought he stood,\n"
        "The Jabberwock, with eyes of flame,\n"
        "Came whiffling through the tulgey wood,\n"
        "And burbled as it came!\n"
        "\n"
        "One, two! One, two! and through and through\n"
        "The vorpal blade went snicker-snack!\n"
        "He left it dead, and with its head\n"
        "He went galumphing back.\n"
        "\n"
        "\"And hast thou slain the Jabberwock?\n"
        "Come to my arms, my beamish boy!\n"
        "O frabjous day! Callooh! Callay!\"\n"
        "He chortled in his joy.\n"
        "\n"
        "'Twas brillig, and the slithy toves\n"
        "Did gyre and gimble in the wabe;\n"
        "All mimsy were the borogoves,\n"
        "And the mome raths outgrabe.\n"
    );

}  // end of setup

void loop () { }

Size:

Binary sketch size: 3,498 bytes (of a 32,256 byte maximum)

RAM left:

Free RAM is 95

Add the F() macro:

size_t freeRam ()
{
  return RAMEND - size_t (__malloc_heap_start);
} // end of freeRam


void setup ()
{
  Serial.begin (115200);
  Serial.println ();
  Serial.print ("Free RAM is ");
  Serial.println ((int)freeRam);
  Serial.println (
      F("  \"Jabberwocky\"\n"
        "\n"
        "'Twas brillig, and the slithy toves\n"
        "Did gyre and gimble in the wabe;\n"
        "All mimsy were the borogoves,\n"
        "And the mome raths outgrabe.\n"
        "\n"
        "\"Beware the Jabberwock, my son!\n"
        "The jaws that bite, the claws that catch!\n"
        "Beware the Jubjub bird, and shun\n"
        "The frumious Bandersnatch!\"\n"
        "\n"
        "He took his vorpal sword in hand:\n"
        "Long time the manxome foe he sought—\n"
        "So rested he by the Tumtum tree,\n"
        "And stood awhile in thought.\n"
        "\n"
        "And as in uffish thought he stood,\n"
        "The Jabberwock, with eyes of flame,\n"
        "Came whiffling through the tulgey wood,\n"
        "And burbled as it came!\n"
        "\n"
        "One, two! One, two! and through and through\n"
        "The vorpal blade went snicker-snack!\n"
        "He left it dead, and with its head\n"
        "He went galumphing back.\n"
        "\n"
        "\"And hast thou slain the Jabberwock?\n"
        "Come to my arms, my beamish boy!\n"
        "O frabjous day! Callooh! Callay!\"\n"
        "He chortled in his joy.\n"
        "\n"
        "'Twas brillig, and the slithy toves\n"
        "Did gyre and gimble in the wabe;\n"
        "All mimsy were the borogoves,\n"
        "And the mome raths outgrabe.\n")
    );

}  // end of setup

void loop () { }
Binary sketch size: 3,570 bytes (of a 32,256 byte maximum)
Free RAM is 572

So we saved (572 - 95) = 477 bytes of RAM at the cost of (3570 - 3498) = 72 bytes of program memory. Sounds like a good trade-off to me.

... at the cost of 704 bytes of Flash memory.

You are dreamin'

I've not compared the code but what I would expect when moving the first string to flash with the F() macro is;

  • The RAM allocation for the string will go away
  • The flash allocation for the string will not change
  • A good chunk of memory will be used up to add the flash access function
  • The amount of flash for the print statement may change slightly because of the new flash enabled function now being used

For each subsequent string moved into flash:

  • The RAM allocation for the string will go away
  • The flash allocation for the string will not change
  • No additional flash memory will be used up because the flash access function is already there
  • The amount of flash for the print statement may change slightly because of the new flash enabled function now being used

The key is that the first string that uses the F() macro will cause a chunk of flash space to be used up for the new flash access code that must be added to the image. But additional F() macro usage won't need to add multiple copies of the code. Then, each print statement that you convert over to reading from flash may change the flash usage by a small amount.

I think the issue with your measurement technique is that you are moving very small strings into flash, and few of them. Moving a one byte string will save you two bytes of RAM (remember the NULL terminator) but may cost you a few extra bytes of flash for the different print function you will call. In addition, if that one byte string was the first time you used a different print function (print() vs println(), for example) you will take the large flash hit again as that new flash access function version is added to your sketch.

I guess my point is that you can't extrapolate the change in flash usage from only a couple strings being moved. You will take an early hit in the beginning, but then less for additional strings.

Also, you are concerned about timing: accessing strings from flash WILL be slower than accessing them from RAM. The Harvard Architecture used by the ATmega family of processors make it more efficient to access code from the code space (flash) and data from the data space (RAM.) That's why strings are copied to RAM in the first place, to put the data in the more efficiently accessed data space. By using the F() macro, you are forcing data to be accessed from the code space, which requires extra work by the processor, and which will take extra time.

There is no free lunch: RAM savings, or fast execution: pick one, you can't have both.

I am attaching the code that I was working with, though I doubt I will be able to come back here. As this is part of my business, I take great offense at being called a “Dreamer”, which is a semi-polite way of saying Liar. Thanks, Nick. Then again, I understand that most are here for “Fun” and I’m a true recluse, so blame me and I’ll just avoid the forum.

The attached copy is the full “Debug” version that I used for all of the tests that I ran. NO other source was used to ensure that all results could be verified as not caused by typo’s, etc. Please ignore my method of coding as I am very old school. I started with binary on a PDP-8M in 1973. This code works as it is, but has not been tuned and every menu entry needs to be corrected and written to EEPROM to allow for useful values. (The first run will be unpredictable.) There is a section of rem’d code to preset these variables, but that should be obvious if reading the code. I didn’t have to run the code to see the compiled size.

There are several pages of documentation and schematics that go along with this, but none of that junk is needed for basic code checking. If anyone wants to make one of these I could post that as well, but connecting to a $250000 CNC without some experience is not something I can suggest. True timing parameters are like this. Parallel input is from the standard Facit 4070 interface which is a parallel tape punch. Parallel output emulates a basic Remex 7300 series tape reader, with a couple other devices included. Both are industry standards in the manufacturing area. I don’t know if it is “legal” to post the actual timing diagrams as they are from the original manufacturer’s specifications, but I could “make” my own version, with disclaimer, and post that, I suppose.

OK, Here is the code. For further information, put a request here and I’ll try to check back here ASAP. I consider the question answered well enough for me to deal with, but other noobs like me might want more info. Sorry for being so long winded. Personal quirk, I supposed.

Serial_BTR.ino (28.3 KB)

Attached are two text files. NoF.txt is a disassembly of the sketch with no use of the F-macro. WithF.txt is a disassembly of the sketch with the use of the F-macro.

The .data section is not included. (The avr-objdump output when the .data section is included is very difficult to read.) String constants (all initialized data) are placed in the .data sections.

NoF.txt (177 KB)

WithF.txt (197 KB)

Thank-You for the listings. They show me three major reasons for all that I was seeing and explain the strange memory allocations. For those that don't want to read the entire code, here is what I learned. 1) The F Macro Does move constant strings from the data area to Code area. These strings Are Not located in the code area of flash memory if the F Macro is not used. They are stored in the data area of the flash. This seems to point to the data area being at the end, like it used to be in older languages. 2) The "weird" amount of allocations are a little more complex to describe, but are obvious if you take general optimization into account. The actual string move section of code uses variables, which, of course, are moved to registers for use. This changes the available registers, which changes what the compiler does. Without stepping through the actual code, line by line, I couldn't give exact examples, but it's very visible in the final assembly code. Once the actual operating code is being compiled, all is almost exactly the same besides addresses, etc. I must assume that the "initialization" code is where these changes occur, which would explain why it's not easy to reason out looking at the source. This code is not part of the written sketch but part of the Arduino requirements for any sketch. I feel a little foolish as I was looking at the wrong code for the reasons. 3) The Actual Data IS stored in the flash, most likely at the end of the code, but must be stored in a different manner. To explain, the "With F" list shows 10266 bytes used, while the program is addressed out to 2804 Hex. This is 10244 bytes of actual code, the remainder must be data. The "Without F" list shows 9572 bytes used, while the code is only addressed out to 1EFA Hex. This is 7930 bytes of actual code. An obvious increase of the data and drastic decrease in code. Hopefully, the implications are just as obvious. I will have to examine the data area to determine the exact differences but I'll bet, as someone else already said, that the extra is required to access the code area directly.

Again, Thank-You for all the attention you have given this, as, while I did look at the final code, I didn't have the foresight nor ambition to save both with and without the F and make that comparison. I hope this information helps others as much as it helps me.

Thank you for the summary of your findings, it makes a lot of sense.

While I know what you mean, and you obviously know what you mean, in the interest of helping others, I'd like to make a small clarification:

Arthur_Clark: 3) The Actual Data IS stored in the flash, most likely at the end of the code

Yes, a copy of the data segment is stored in flash, but the code does not actually access the data from there. For example, if the code contains a line like:

char hello[] = "Hello World!";

Then there will be an area of 13 bytes of RAM allocated for the variable hello: 12 characters of the string, plus a NULL terminator. This is allocated in RAM because that's where the processor can access it most efficiently. But RAM does not remember anything while the power is off, so the characters "Hello World!" must be stored in flash memory somewhere. This is where the data segment in flash comes in. When the sketch starts up, it runs a bit of initialization code before setup() is called, and one of the things it does is copy the data segment from flash to RAM so that the RAM variables have their starting values.

As this subject also evolved into memory space used by repetitive F() macros.

An alternative to the F() is to allocate the flash memory strings "once" using:

const char MY_STRING[] PROGMEM = "A Prog Memory Resident String";

, and then dependning on C-skills copy data to a buffer using strcpy_P()

Being a bit undisciplined and focused on in-house projects, length checking has been ignored.

I tried something like this:

//////////////////////////////////////////////////////////////////////////
// Description  : Return progmem string as null terminated string 
//
// Return       : char* referencing the copied string 
// Usage        : Serial.Print(f(MY_STRING))
// MY_STRING declared as:
//  const char MY_STRING[] PROGMEM = "A Prog Memory Resident String";  
//////////////////////////////////////////////////////////////////////////
char* f(const char* text) 
{
 static const char buf[64] ={'\0'};
 return strcpy_P(buf, (char *) text);
}

,so printing a memory string to a stream is as simple as:

Serial.println(f(MY_STRING));

At least this is what I ended up using and critical RAM usage of 93% droped to 80% just moving some of my ten longest URL;s to flash.

(If I use strings longer than 64 bytes I automatically get reminded not to speak to much. And to stream data in smaller portions)

???

While I was dwelling on the bug I was inspired by this streaming example and made another alternative with no buffering. Probably have to dwell on that too?:

//////////////////////////////////////////////////////////////////////////
// Description  : Write flash resident string to stream
//
// Return       : nothing
// Usage        : streamf(Serial, MY_STRING)
// MY_STRING declared as:
//  const char MY_STRING[] PROGMEM = "A Prog Memory Resident String";
//////////////////////////////////////////////////////////////////////////
void writef(Stream &port, const char* text)
{
 char c;
 if(!text) return;
 while((c = pgm_read_byte(text++))) {
 port.write(c);
 }
}

And used it like this:

writef(Serial, MY_STRING);

flodis: Being a bit undisciplined and focused on in-house projects, length checking has been ignored.

That is not the only thing you ignored. You missed it by one keyword.

For everyone else, the code has a serious bug.

[quote author=Coding Badly link=msg=3482896 date=1510427876] That is not the only thing you ignored. You missed it by one keyword.

For everyone else, the code has a serious bug.

[/quote]

I think I fixed it now nailing the buffer to the wall with a 'static'

I think I fixed it now nailing the buffer to the wall with a 'static'

-_-

There is no reason to copy the PROGMEM array to RAM or to make function that duplicates existing functions. Just print it from PROGMEM by casting the argument to the correct type:

#define CF(p) ((const __FlashStringHelper *)p)

const char MY_STRING[] PROGMEM = "A FLASH String";

   ...

  Serial.print( CF( MY_STRING ) );

-dev:
-_-

There is no reason to copy the PROGMEM array to RAM or to make function that duplicates existing functions. Just print it from PROGMEM by casting the argument to the correct type:

#define CF(p) ((const __FlashStringHelper *)p)

const char MY_STRING PROGMEM = “A FLASH String”;

Serial.print( CF( MY_STRING ) );

Perfect!

I tested to pass a reference to functions that want char* pointers and got :

no known conversion for argument 1 from const __FlashStringHelper* to const char*

I made a brute cast to char* in the suggested macro:

#define CFx(p) (const char *)((const __FlashStringHelper *)p)

What is the appropriate way?

flodis: What is the appropriate way?

I cannot speak for anyone else but I have no idea what you are trying to do.

So, yet again...

Post your code.

flodis:
I tested to pass a reference to functions that want char* pointers and got :

no known conversion for argument 1 from const __FlashStringHelper* to const char*

You can’t pass a “__FlashStringHelper *” to a function that takes a “char *”. You have to create a separate version of the function that takes a “__FlashStringHelper *” and does the fetches from FLASH.

All of the gory details:

On the AVR processor the SRAM, FLASH, and EEPROM are in separate address spaces. Instructions are fetched from FLASH (a.k.a. Program Memory or PROGMEM). Data is read from and written to SRAM. You have to use special machine instructions for reading FLASH or reading and writing EEPROM. Any initialized variables will be initialized by copying the data from FLASH to the variable’s location in SRAM before the sketch starts. This is great for initialized variables that will be modified by the sketch but for constant (read-only) variables it is a waste of precious SRAM space. For larger initialized read-only variables, such as lookup tables, it is good to tell the compiler to keep the data in FLASH.

Unfortunately the compiler will only keep track of the address of the variable, not which address space it is in. It won’t remember that your variable is in FLASH and if you try to use it like you would any other variable it will use that FLASH address to fetch data from SRAM and get the wrong data. It is up to you to add the special function calls to fetch data from FLASH whenever you want to fetch data from that variable.

String literals are a special case of read-only data. They are treated like initialized read-only variables and their space in SRAM is initialized by copying the data from FLASH. Because they are often used for text output to Serial, LCD, Ethernet, WiFi, etc. there is a special trick used to make keeping them in FLASH easier.

There is a macro named “F” in the standard include file Wstring.h:

class __FlashStringHelper;
#define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))

The F() macro uses the macro PSTR() to tell the compiler to keep the string in FLASH and then changes the value type from ‘char *’ (character pointer) to a ‘__FlashStringHelper *’. The class __FlashStringHelper has no body, just a type. The “reinterpret_cast” tells the compiler that you know that the value being cast is not compatible with the destination type. Unlike a regular cast, no conversion is done. The ONLY safe operation is to cast the value BACK to what it was before.

In the Print class there are .print() and .println() methods similar to the ‘char *’ methods that accept a ‘__FlashStringHelper *’ instead. The compiler chooses those methods when you pass the ‘__FlashStringHelper *’ created by the F() macro. The methods then cast the __FlashStringHelper pointer BACk to a character pointer and fetch each character of the string from FLASH. Serial, LCD, Ethernet, WiFi, etc are all objects that inherit behavior from the Print class so their .print() and .println() methods can all accept the ‘__FlashStringHelper *’ type and fetch the string from the FLASH address space.

If you write your own function and one of the arguments is often a fairly large string literal you might want to make a version of the function that accepts the ‘__FlashStringHelper *’ type. Then you can use the F() macro to keep those string literals in FLASH. See Print.cpp in the Arduino core sources for examples of how you would treat the argument differently.

You can also use the macro when initializing a table of strings:

__FlashStringHelper *myStrings[] = {F("abcdefg"), F("xbcdefg")};

void setup(void) {
  Serial.begin(19200);
}
void loop(void) {
  Serial.println(myStrings[0]);
  Serial.println(myStrings[1]);
  delay(1000);
}

johnwasser:
You can also use the macro when initializing a table of strings:

__FlashStringHelper *myStrings[] = {F("abcdefg"), F("xbcdefg")};

void setup(void) {
 Serial.begin(19200);
}
void loop(void) {
 Serial.println(myStrings[0]);
 Serial.println(myStrings[1]);
 delay(1000);
}

Unfortunately you can’t use it for global variables, F() and PSTR() must be inside a function:

.../WString.h:38:74: error: statement-expressions are not allowed outside functions nor in template-argument lists
 #define F(string_literal) (reinterpret_cast<const __FlashStringHelper *>(PSTR(string_literal)))
                                                                          ^
.../fsh.ino:1:37: note: in expansion of macro 'F'
 __FlashStringHelper *myStrings[] = {F("abcdefg"), F("xbcdefg")};

oqibidipo: Unfortunately you can't use it for global variables, F() and PSTR() must be inside a function.

Darn! I was sure I had tried it and it worked. This version is not nearly as neat but it does compile:

const __FlashStringHelper *myStrings[] = {
  (const PROGMEM __FlashStringHelper *) "abcdefg", 
  (const PROGMEM __FlashStringHelper *) "xbcdefg"};

[quote author=Coding Badly link=msg=3483248 date=1510449664] I cannot speak for anyone else but I have no idea what you are trying to do.

So, yet again...

Post your code.

[/quote]

Question:

If we have one string in RAM and one in flash memory:

const char hello1[] = "Hello World!";
const char hello2[] PROGMEM = "Hello World!";

And a function

size_t fun(const char *)
{
    //Some code..
}

You can call fun() as:

size_t myvalue = fun(hello1);

How do you call fun() using hello2?

From the compiler's perspective, hello1 and hello2 are the same datatype. The purpose of __FlashStringHelper is to give PROGMEM data a different datatype so the compiler can treat them differently.

In other words, you call fun() using hello2 by passing it is a parameter...

  fun( hello2 );

But that is not very useful because it is impossible to treat hello1 and hello2 differently in fun because the compiler, as I said earlier, sees the two as the same datatype.

flodis: You can call fun() as:

size_t myvalue = fun(hello1);

How do you call fun() using hello2?

Easy! size_t myvalue = fun(hello2);

The problem is that fun() has to KNOW if the address being passed is in RAM or PROGMEM. By default, all character pointers are assumed to be pointing to RAM. If fun() doesn't take pains to fetch data from PROGMEM it will be fetched from RAM and only 'hello1' will produce the expected results. If fun() fetches data from PROGMEM, then only 'hello2' will produce the expected results.