Confused over operation of strlcpy, strlcpy_P & strlcpy_PF

Projects Programming
21

[...continued...]

7) DIFFERENCE BETWEEN FUNCTION USING RAM AND FLASH
The function that accepts something of datatype __FlashStringHelper will make use of:

byte b = pgm_read_byte (FLASH_16bitaddress);

within the function to gain access to FLASH data instead of RAM data (the default). This function works for all AVR processors PROVIDING THAT THE FLASH DATA IS IN THE FIRST 64KB. As already noted for the Mega2560 this may not be the case. This 64KB limit arises because __FlashStringHelper is a 16 bit address in effect.

8) FLASH USE ABOVE 64KB PROBLEMATIC
The use of FLASH memory above 64KB is problematic with many libraries. This arises because it is only the MEGA2560 processor that has provision to store data above 64KB. For this reason it is necessary to ensure that both compiled code and PROGMEM use does not cross the 64KB boundary as unexpected operation will occur with many libraries. This failure will not be flagged to the programmer in any way and will result in data in FLASH being accessed at the wrong FLASH address. How your program deals with this will be a can of worms depending on the use that is made of the data (it may be stuff other than text arrays).

9) MEGA2560 FLASH ISSUES
This unexpected operation with the Mega2560 arises because references to memory (both RAM and FLASH) have the compiler using 16 bit pointers by default. To operate with a Mega2560 a 18 bit pointer (address specifier) is needed. This has no practical datatype representation in the AVR so 32 bits get allocated to handle this. This 32 bit pointer to a FLASH address (above 64KB) is typically given the datatype:

uint_farptr_t

in practice this is just a redefine of uint32_t:

#typedef uint32_t uint_farptr_t; (in inttypes.h)

When using data stored in FLASH above 64KB (on the Mega2560) one needs to use the uint_farptr_t datatype in conjunction with the function:

pgm_get_far_address (16 bit address)

This essentially takes a 16 bit address and returns a 32bit value (of type uint_farptr_t). The pgm_get_far_address performs the necessary translation from 16 bit to 18 bit address for Mega2560. When working with the Mega2560 you need to essentially do two things:

a) You need to use pgm_get_far_address (16 bit address) in conjunction with
b) pgm_get_byte_far (32 bit address)

to have things work. So pgm_get_far_address and pgm_get_byte_far really go hand in hand and need to be used together.

10) LIBRARY SUPPORT FOR MEGA2560 ADDRESSES
These 32 bit address (18 bits effective) on the Mega2560 can be used with some of the string related functions like strlcpy (string copy) and strlcat etc. The trick here is to use the _PF variant so:

use strlcpy_PF (dest_string, source_string_in_flash_32_bits_in_size, max_size) instead of strlcpy (…)
use strlcat_PF (…) instead of strlcat.

11) LIBRARY _P VARIANTS
There are also _P variants of these functions (eg. Strlcat_P) and these only work with FLASH addresses located in the first 64KB (16 bit addresses). On a Mega2560 you wont be advised if you go over 64KB so in my view better to use the _PF variant to be safe - with pgm_get_far_address - to provide code that will work no matter where the FLASH data is located.

12) LIBRARY WORK AROUNDS ON MEGA2560
Because of the complexities associated with Mega2560 processors the following code would be typical of using strlcpy_PF where the source data could be located above 64KB:

//Global stuff...
const MaxStringSize = 100;
const char GJB_FlashData[] PROGMEM = “Yep, Im in FLASH and could be above 64KB depending on code size and other PROGMEM use.”;

//Later in in code...

loop () {
//Other stuff...

char tString[MaxStringSize];
strlcpy_PF (tString, pgm_get_far_address (GJB_FlashData), MaxStringSize);
...
}

In other words when using the _PF variants of library functions it is also necessary to use pgm_get_far_address() as well. The resultant tString (in RAM) could then be used with Serial.print () and all other library functions (strlcat, strtok etc). Serial.print and other functions, like many other library functions, silently fail when dealing with addresses above 64KB. The higher order bits (above the first 16) are silently thrown away so you will have fun if you don’t factor this into AVR operation. Using this technique everything is transferred from FLASH (whereever it resides) to RAM where life is easy.

13) F() AND FUNCTIONS WITH SUFFIX _PF
I initially thought that those library functions ending with _PF (eg. strlcpy_PF) were for use with the F() macro. For example:

char tString[MaxSize];
strlcpy_PF (tString, F(“I am in FLASH somewhere…”), MaxSize);

The use of F() has NOTHING to do with the _PF variants. The F in the _PF means FAR – ie. above 64KB. As an aside the F() macro I believe only works if located in the first 64KB for similar reasons already given – if located above 64KB the additional bits of the address are not factored in and FUN returns. For example:

Serial.print (F(“I will be displayed if located in first 64KB.”));

will only work if the chars are located below 64KB and this is because the Serial.print function cannot handle addresses above 16 bits.

14) FINAL COMMENT
There is a lot to learn to make use of the extra FLASH the Mega2560 offers. The traps are numerous and can occur in several places. One can expect a small performance loss when accessing FLASH above 64KB. Using the Mega2560 for the extra RAM it provides (8KB compared to 2KB for most other AVR processors) is easy. Using the extra FLASH is not easy.

I would like to thank earlier contributors to my understanding - particularly J-M-L and david_2018.

Geoffrey

22

Oh. While getting a handle on all of this I came across the following summary of the strlcpy and related string function summary which was good for the details about data types as well as the plain explanation of its operation:

http://ugweb.cs.ualberta.ca/~c274/resources/arduino-ua/avr-libc-1.7.1-overlay/avr-libc/avr-libc-user-manual-1.7.1/group__avr__pgmspace.html

Geoffrey.

23

@Geoffrey - +karma for a great summary of all your findings. That post will help many and possibly would deserve a dedicated post with an appropriate title to make it easy to find.

Very well done !!!

24

Excellent write-up!

A note about point #2. Not only does the char array need to be CONST, but it also needs to be STATIC. Global variables are static by default so that keyword is not required. If you attempt to declare a string in PROGMEM inside a function (non-global) it will not work. That is the reason behind the PSTR macro including the keyword 'static' since it makes it universally work.

In point #8 (or smiley face) you state

For this reason it is necessary to ensure that both compiled code and PROGMEM use does not cross the 64KB boundary

but compiled code (e.g. instructions) have nothing to do with this topic. You can write a massive program that consumes all of flash and it will work just fine without any special considerations.

25

One question which relates to all of this and which still is not understood by me is this #define:

#define pgm_get_far_address(var)                          \
({                                                    \
	uint_farptr_t tmp;                                \
                                                      \
	__asm__ __volatile__(                             \
                                                      \
			"ldi	%A0, lo8(%1)"           "\n\t"    \
			"ldi	%B0, hi8(%1)"           "\n\t"    \
			"ldi	%C0, hh8(%1)"           "\n\t"    \
			"clr	%D0"                    "\n\t"    \
		:                                             \
			"=d" (tmp)                                \
		:                                             \
			"p"  (&(var))                             \
	);                                                \
	tmp;                                              \
})

I understand that if you have something like:

//Global stuff...
#define PROGMEM_LATE1 __attribute__ (( __section__(".fini1") ))
const char GJB_Stuff1[] PROGMEM_LATE1 = "This is some text in PROGMEM_LATE1.";

//In loop...
loop () {
  ...
  Serial.print (F("|ADDR (GJB_Stuff1 pgm_get...) ")); 
  Serial.print (pgm_get_far_address (GJB_Stuff1), HEX); 
  Serial.print (F("|"));
  ...
}

Correctly displays an address that is above 64KB:

|ADDR (GJB_Stuff1 pgm_get...) 11A48|

But when you display the size of &GJB_Stuff1 you get:

2 bytes

So pgm_get_far_address does some voodoo to translate a 16 bit address to 32 bits (of which 18 are only relevant on Mega2560). It does this with the assembly code already provided.

What I don't get is that the assembly takes:

"p" (&(var))

as its input data and the assembly code copies the lowest 8 bits (lo8), the next higher 8 bits (hi8) and the next higher 8 bits (hh8) into tmp (a 32 bit thing). This tmp (32 bits) is then returned.

Given that GJB_Stuff1 address is only 16 bits there is some magic voodoo going on. I suspect that the code is somehow bypassing the compiler limitations (16 bit) and accessing the true address that the linker (maybe) knows about.

It seems that the "window" the compiler provides to flash addresses is bypassed and the only thing I can think of is that the linker translates &var into a correct 18 bit address and loads %A0, %B0 & %C0 correctly - maybe the "p" specification forces this to happen.

If anyone can explain this bit of assembler operation it would be much appreciated.

Geoffrey.

26

In answer to david_2018 who wrote:

Your code is very similar to that used in Print.h, with a slight difference that you use Serial.print verses Serial.write to actually print the character, and the Print.h code returns the number of bytes written, something that is seldom actually used but should be done to maintain compatibility.

It should be possible to define something similar to __FlashStringHelper* to handle far addresses, but I doubt there has been enough need for anyone to spend time doing it, since in the few instances it is needed a custom function such as yours can be written.

You may run into other problems storing massive amounts of text strings above the 64K boundary, unless you intend to reference each explicitly. Storing an array of char* to the text strings runs into the problem of the compiler treating a pointer as a 16-bit value, and the compiler cannot resolve the 32-bit address at compile time either

I think I understand what you mean and perhaps this example will clarify. If we have:

//Globals...
const char* a_dict PROGMEM = "a";
const char* aa_dict PROGMEM ="aa";
const char* aaa_dict PROGMEM ="aaa";
const char* aaron_dict  PROGMEM ="aaron";
const char* ab_dict PROGMEM ="ab";
const char* abandoned_dict PROGMEM ="abandoned";
const char* abc_dict PROGMEM ="abc";

const char* dictionary[] PROGMEM = {a_dict, aa_dict,aaa_dict,aaron_dict,ab_dict,abandoned_dict,abc_dic, ...};

Then if the data in PROGMEM goes above the 64KB barrier then the addresses stored "in" dictionary are all too short (16 bits). The only way to fix is to use pgm_get_far_address (a_dict) etc and this cannot be used in the definition of the dictionary array - ie. this will NOT work:

uint_farptr_t dictionary[] PROGMEM = {pgm_get_far_address(a_dict), pgm_get_far_address(aa_dict), ...};

Additionally, what I believe you are also saying is that using:

int i;
uint_farptr_t BigAddress;
uint_farptr_t BigAddressofString;
for (i = 0; i < somenumber; i++) {
   BigAddress = pgm_get_far_address (&dictionary[i]);
   BigAddressofString = pgm_get_far_address(pgm_get_word_far (BigAddress)); //I fail.
   GJBserialOutputUsing32bitPtr (BigAddressofString);
  ...
}

Will fail as once we have moved away from a 1D array (the array or chars - eg a_dict) there is an inability to use pgm_get_far_address on the data from the use of the first index into a 2D array to get to the real location of the character array in FLASH.

Is that correct ?

Geoffrey.

27

blh64 wrote:

A note about point #2. Not only does the char array need to be CONST, but it also needs to be STATIC. Global variables are static by default so that keyword is not required. If you attempt to declare a string in PROGMEM inside a function (non-global) it will not work. That is the reason behind the PSTR macro including the keyword 'static' since it makes it universally work.

Have learnt this the hard way and yes understand that. The keyword "static" in effect makes it "live" for the entire time the program is running when the PROGMEM is used inside a function. It can only be accessed inside the function so is like a globally defined value only accessible in one function. The PROGMEM MUST exist always in a program.

Thanks for clarifying.

Geoffrey.

28

blh64 wrote:

From my summary:
In point #8 (or smiley face) you state:

For this reason it is necessary to ensure that both compiled code and PROGMEM use does not cross the 64KB boundary.

Response:

but compiled code (e.g. instructions) have nothing to do with this topic. You can write a massive program that consumes all of flash and it will work just fine without any special considerations.

The context here was in relation to using Arduino libraries. In this context, as best I can see, if we have program code that is large enough to force PROGMEM defined data, F() data or PSTR() data above the 64KB limit then issues WILL occur. For example Serial.print () accepts something of type __FlashStringHelper but while the F() macro is natively of this datatype one can typecase the PROGMEM or PSTR into this as well. In any event if program code has forced data in FLASH above the 64KB boundary issues will occur because Serial.print is not >64KB FLASH aware.

I believe that what you are referring to is that if you write code that is really large and goes above 64KB and does NOT use any libraries then the execution of the code itself will not have any problems - in other words the compiler and operation of the PC (program counter) work when code goes >64KB.

This, if this is what you are referring to, is a edge case and I think may be missing the context that I was getting at. IF you use Arduino libraries then the functionality of data stored in FLASH is lost when it goes above 64KB is all cases (libraries do not factor this in). The amount of storage you have for FLASH data is a function (when libraries are used) of not crossing the 64KB boundary and therefore the machine code size impacts this as it is located, by default, starting at low FLASH addresses.

Phew !

Geoffrey.

29

The context here was in relation to using Arduino libraries. In this context, as best I can see, if we have program code that is large enough to force PROGMEM defined data, F() data or PSTR() data above the 64KB limit then issues WILL occur.

Program code itself will not force PROGMEM defined data above the 64KB boundary, because the compiler organizes the flash memory such that the code is always placed after the PROGMEM data, unless specifically told to do otherwise. What gets you in trouble is when the total amount of data being stored in PROGMEM exceeds the 64KB boundary. Using libraries has nothing to do with it, unless you use multiple libraries that use enough total PROGMEM to exceed this amount, and that is rather unlikely. The problem with exceeding 64KB usually occurs when your code stores a massive amount of data in PROGMEM, because that can push other PROGMEM data to an address above 64K, data that is assumed to always be at a lower address. For that reason, you specifically need to tell the compiler to place your data after the code instead of before it, leaving the remaining PROGMEM data where it will work properly. Since you know that your data exceeds the 64KB boundary, you will then be able to handle it properly.

//Globals...
const char* a_dict PROGMEM = "a";
const char* aa_dict PROGMEM ="aa";
const char* aaa_dict PROGMEM ="aaa";
const char* aaron_dict PROGMEM ="aaron";
const char* ab_dict PROGMEM ="ab";
const char* abandoned_dict PROGMEM ="abandoned";
const char* abc_dict PROGMEM ="abc";

const char* dictionary[] PROGMEM = {a_dict, aa_dict,aaa_dict,aaron_dict,ab_dict,abandoned_dict,abc_dic, ...};

That is an example of how NOT to store text in PROGMEM. The char* to the text is stored in PROGMEM, but the text itself will be stored in ram. This is how it should be done:

//Globals...
const char a_dict[] PROGMEM = "a";
const char aa_dict[] PROGMEM ="aa";
const char aaa_dict[] PROGMEM ="aaa";
const char aaron_dict[]  PROGMEM ="aaron";
const char ab_dict[] PROGMEM ="ab";
const char abandoned_dict[] PROGMEM ="abandoned";
const char abc_dict[] PROGMEM ="abc";

const char* const dictionary[] PROGMEM = {a_dict, aa_dict,aaa_dict,aaron_dict,ab_dict,abandoned_dict,abc_dict};

You are correct that this will not work:

uint_farptr_t dictionary[] PROGMEM = {pgm_get_far_address(a_dict), pgm_get_far_address(aa_dict), ...};

pgm_get_far_address() cannot be resolved by the compiler, it is a function that must be executed at run-time, and the address that it returns is only know after the code is run through the linker, which occurs after compilation completes. You can create the dictionary array in ram at run-time, but that results in a large amount of code and for a large dictionary may exceeds the limits of ram.

int i;
uint_farptr_t BigAddress;
uint_farptr_t BigAddressofString;
for (i = 0; i < somenumber; i++) {
   BigAddress = pgm_get_far_address (&dictionary[i]);
   BigAddressofString = pgm_get_far_address(pgm_get_word_far (BigAddress)); //I fail.
   GJBserialOutputUsing32bitPtr (BigAddressofString);
  ...
}

A few problems with this code, ignoring the fact that the dictionary array was not able to be created in PROGMEM. The compiler cannot resolve the reference to dictionary[ i ] unless it is stored within the lower 64KB of PROGMEM, because it does address calculations using 16-bit numbers. You will need to get the address of dictionary first, then add the offset to element i, such as

BigAddress = pgm_get_far_address(dictionary) + i * sizeof(dictionary[0]);

Another problem is that you are reading a word from the dictionary array, which is two bytes, but the addresses stored in the array are four bytes each, so you need to use pgm_read_dword_far().