Is String safe to use on a ESP8266?

Oh boy this reply got quite long!!! :o :o :o

Robin2: The problem is the word "maybe" :)

Without trying, I can't be sure of anything.

DKWatson: Enhance your skills and develop (or Google) macros and/or functions that provide similar functionality. In fact you can get carried away and end up with a library that offers capability beyond that of the String class.

Yeah I could, but is not my goal right now. Anyway, I bet there are already some libraries that extend this funcionality, as you said.

PieterP: Using less intuitive cstring functions does not make you a better programmer.

Not better, but more experienced.

My definition of "better programmer" is the one who knows how to apply all the good practices of programming. Doesn't need to create perfect code at the first time; even large software develop companies have to release patches/updates to fix some stray bugs or oversights that couldn't be noticed no matter how much they tested.

PieterP: If you reserve enough memory beforehand, and you see to it that not too many intermediate Strings are created, you won't run into trouble, especially if you have such a large heap as on the ESP.

Probably that is, but who knows... real life is never that easy.

Danois90: if I would venture into using the String class, I would program my sketch to monitor the amount of free, dynamic memory very carefully. If at any point the available memory drops below 5-10%, I would "alarm the user" somehow.

This could work fine if fragmented free space is also counted. The method usually looks for the furthest byte (position) ever allocated to something, but doesn't take in account whatever is in between (e. g. destroyed objects creates free space, if it was between other data, then it's fragmented free space).

PieterP: The amount of free memory doesn't tell you anything about the fragmentation level.

That's exactly what I said just before.

J-M-L: for example when you use + (concatenate) operations that will allocate a temporary StringSumHelper - even if you used plenty of pre-allocated space for the function.

So C-strings are safer anyway?

J-M-L: So you have very limited control on what's really happening... and long running system will possibly be impacted over time, which drives developer to handle this by rebooting their system from time to time to clean up...

That's what I'm afraid of...

J-M-L: so all bets are already off if you rely on those libraries...

I have planned to use mostly some of the core libraries (Wi-Fi controller, TCP server, serial controller for the Bluetooth terminal/console, UDP client if I plan to use NTP for logging purposes), and another one for password storage with SHA-256 hashing (probably I'm gonna use the virtual EEPROM or files from the internal SPFS). Hope I'm not in trouble already.

Danois90: If the amount of free heap is reported as 20k, and you cannot allocate a buffer of 2-5k, that would indicate that fragmentation may become an issue.

Then it's matter of testing it out.

Although I'm thinking: if objects are constanly created but inmediately destroyed (let's say because of how String concatenation works), fragmentation shouldn't occur as long as somehow another object or array isn't allocated outside of the function's code (i. e. due to an interrupt).

Robin2: If that happens is there anything you can do to fix it, apart from re-booting?

Danois90: I do not think there is any feature to "defragmentate" memory.

Also called "compaction", it's the last stage of garbage collecting. Since that implies moving entire blocks of data, the process becomes pretty time-consuming; and also will block the program execution (even disable interrupts to avoid race condition) if no multitasking capability is available.

Danois90: But knowing (in code) that there is a problem evolving, makes it possible to issue a reset before the device locks up.

In code, only if you can tell me that the restart() function from the ESP SDK (available in the Arduino IDE as well, inside an object called "ESP"), provokes an actual "reboot" rather than a simple "soft-reset".

Another solution is some kind of hardware watchdog timer; you know, the timer that triggers a reset if it isn't restarted after certain timeout. When the execution freezes for whatever reason, the microcontroller is no longer able to restart that timer; thus eventually this watchdog kicks in and triggers the same reset as if you pushed the button.

Does the ESP8266 have one? If it does, then I'd have no more to worry about.

econjack: It seems to me your coding style is geared to working around an issue that you know exists even though a tighter approach mitigates the problem altogether. It makes even less sense since you know how to use C strings and, thus, avoid the problem. Why would you do that?

On an ATmega328P I have no doubt that C-strings are the way to go; however, in an enviroment with more RAM like the ESP8266, I might give myself the luxury of using String instead, without compromising reliability. The point of this thread is to discuss if that last (bolded) statement is true or not.

Robin2: The String class can bite you even if the program is perfectly written

That's why I have such negative preconception with String.

wildbill: If you make use of Strings, even with this much RAM, you're gambling that everything will be ok. If this is a 24/7 application, that gamble may not bite you for days, so you're never certain that your testing was enough. In the end, it comes down to how critical the application is and what the consequences will be if it crashes - you just have to decide whether to accept the risk.

So... I guess I can take it. The application isn't that critical, but an unresponsive device is annoying nonetheless.

econjack: I was arguing that he sees them as the solution to a problem that can be avoided by other, less risky, alternatives. His alternative is a "work-around" for a known problem when a simple, viable, alternative that he knows how to code and understands, exists.

And you are right.

But the project isn't that big anyway (unless I decide to make it commercial); so that's why I am not necessarily looking for optimization but simplicity and workarounds for any possible caveat.

On an ATmega328P I have no doubt that C-strings are the way to go; however, in an enviroment with more RAM like the ESP8266, I might give myself the luxury of using String instead, without compromising reliability. The point of this thread is to discuss if that last (bolded) statement is true or not.

Well if you are already convinced your memory gets fragmented in a 328p, then you know it’s just a matter of time before you get bitten in an ESP... it will just happen, likely later.. so you are just buying yourself more time - which might be good enough if your device does not run for too long without a reboot... but good enough does not mean reliable...

A String only reallocates memory when it gets larger than its buffer (e.g. due to concatenation). If you reserve enough memory beforehand, and if you do not create unnecessary copies of Strings, you will be fine.

Fragmentation occurs when a String grows, because the new buffer is allocated before the old one is freed. Intermediate Strings are also not an issue, as long as they are freed before malloc is called for an object that has a longer lifetime than these Strings.

Sure if you know what you do you can be on the safe side.... but that’s a lot of if or as long as - which are not friendly for beginners and require understanding of what the library does (If I remember correctly Strings are allocated by blocks of multiple of 16 bytes) and how temporary objects are disposed of;

I have not tested but If you do as = String(“IP(“) + WiFi.localIP().toString() + “) is how you find me”; even If s has plenty of pre-allocated space you will create a temporary hole Since the IP buffer resulting from the function call will only be cleared at the end of your encompassing function. So once executed you will have a 16 bytes hole and the IP String buffer (16 bytes).

==> you can’t control a LIFO allocation/deallocation to keep the heap clean

==> if that happens when you are short in memory and need a bigger block then you’ll have weird issues...

Note that in that example the memory allocation scheme will vary depending on the length of the IP. Whilst “IP(192.168.0.5” would fit in the default 16 bytes allocation, “IP(192.168.0.122” would overflow the first temporary String buffer allocation (the one for “IP(“) and thus a new 32 bytes buffer would be reallocated after the IP address String buffer leaving a 16 byte hole that cannot be used for concatenating the “) is how you find me”which will be put in its own extra 32 byte dynamically buffer (which might not be available)

Lucario448:

The problem is the word "maybe"

Without trying, I can't be sure of anything.

I was trying to suggest that you won't be certain even after carrying out experiments.

...R

My experience : there is no problem if (using Strings globally) you get a memory usage <70%.

In fact, the only problem I encountered with Strings so far was one project processing a sms receiver on an uno. When compiler “said” “global variables use (<~1450)” everything was ok. After adding some variables and usage exceeded ~1520…crash.
But, an sms String is a very - very big String and extracting data out of it means too much extra memory need. Use of 20 Strings, with a length of 10-20 each, and 500more bytes in variables…I dont think that they can create a problem in any way.
Except, if user codes

 String string21=string1+string2......+string20;

Dont blame “Strings” in such case…

You just can’t be sure and pray for things to go well...

J-M-L: You just can’t be sure and pray for things to go well...

Then I saw THEIR face, now I'm a believer :)

edit : or better Then I saw String's face, now I'm a believer Not a trace, of doubt in my mind I'm in love, and I'm a believer I couldn't leave Them if I tried :):):)

The real nightmare with Strings, for me at least, is that they can take away one of your best debugging tools, namely "what changed last?".

Some months from now, when your String infested program has been running just fine, you decide to add some new features for once perhaps not involving new Strings. Then your system starts acting up and you assume that the new features are the cause and try to debug them. But it turns out that the real cause is that you consumed memory that the heap was using for Strings and now fragmentation has occurred. Strings not only bit you, but made it harder to debug.

So to paraphrase Dr. Ian Malcolm: "Strings: just because you could doesn't mean that you should".

Hmmm Although the following is not straight related to subject, it is about Strings so I give it a try.

//in the beginning

String TestString="abcd......all throu xyzabc ...again.." ; //let say tottaly 4 times=4*26=104+2=106 bytes 


//no change of TestString

//then in a function

TestString="";

what of the following is correct? The memory space that was kept for TestString a. is free now for any other use or b. it keeps "occupied" for use of TestString (look for free space for TestString is done only if TestString increases more than 106 bytes)

The answer is in the source code

for your global, the constructor is

String::String(const char *cstr) {
    init();
    if(cstr)
        copy(cstr, strlen(cstr));
}

with init being

inline void String::init(void) {
    buffer = NULL;
    capacity = 0;
    len = 0;
}

so it calls

String & String::copy(const char *cstr, unsigned int length) {
    if(!reserve(length)) {
        invalidate();
        return *this;
    }
    len = length;
    strcpy(buffer, cstr);
    return *this;
}

which reserve enough space for the length of your string (actually reserve will reserve a bit more) as you can see here

unsigned char String::reserve(unsigned int size) {
    if(buffer && capacity >= size)
        return 1;
    if(changeBuffer(size)) {
        if(len == 0)
            buffer[0] = 0;
        return 1;
    }
    return 0;
}

which will call

unsigned char String::changeBuffer(unsigned int maxStrLen) {
    size_t newSize = (maxStrLen + 16) & (~0xf);
    char *newbuffer = (char *) realloc(buffer, newSize);
    if(newbuffer) {
        size_t oldSize = capacity + 1; // include NULL.
        if (newSize > oldSize)
        {
            memset(newbuffer + oldSize, 0, newSize - oldSize);
        }
        capacity = newSize - 1;
        buffer = newbuffer;
        return 1;
    }
    return 0;
}

which is the piece doing the realloc() and then if that worked, copy your cString into the buffer.

then when you perform your TestString=""; it calls

String & String::operator =(const char *cstr) {
    if(cstr)
        copy(cstr, strlen(cstr));
    else
        invalidate();

    return *this;
}

which calls copy with an empty cString (length will be 0). we have seen that copy calls first reserve(0) and thus the first part of the reserve function will be triggered

if(buffer && capacity >= size)
        return 1;

and no changeBuffer() will be called and then your empty string is duplicated into the buffer.

→ so the memory stay occupied

By the way your assumption that it is 106 bytes is wrong as the size passed for the malloc() is (maxStrLen + 16) & (~0xf);, so the next 16 byte buffer that is big enough for your string

if you want to see what this formula does run this

void setup() {
  Serial.begin(115200);
  for (unsigned int i = 0; i<120; i++) {
    Serial.print(i);
    Serial.print(F("\t\t"));
    Serial.println((i + 16) & (~0xf)); // closest bigger 16 multiple, I would have done (((i >> 4) +1) << 4) not sure if it's faster
  }
}

void loop() {}

Now if your String was not a global then it’s scope is limited and the memory will be claimed when the objects is no longer needed.

J-M-L:
By the way your assumption that it is 106 bytes is wrong as the size passed for the malloc() is (maxStrLen + 16) & (~0xf);, so the next 16 byte buffer that is big enough for your string

Read about 16-byte blocks some posts above. Error in “the run”.

J-M-L:
if you want to see what this formula does run this

void setup() {

Serial.begin(115200);
 for (unsigned int i = 0; i<120; i++) {
   Serial.print(i);
   Serial.print(F("\t\t"));
   Serial.println((i + 16) & (~0xf)); // closest bigger 16 multiple, I would have done (((i >> 4) +1) << 4) not sure if it’s faster
 }
}

void loop() {}

Yes. Done.
thank you so much

J-M-L:
Well if you are already convinced your memory gets fragmented in a 328p, then you know it’s just a matter of time before you get bitten in an ESP… it will just happen, likely later… so you are just buying yourself more time - which might be good enough if your device does not run for too long without a reboot…

That’s a good point.
However, I’m also thinking if RAM is plenty, it still can get fragmented; but eventually those “holes” get big enough that the space essentially becomes self-sustainable. That is, unless some insane allocations don’t let these “holes” grow enough.

PieterP:
A String only reallocates memory when it gets larger than its buffer (e.g. due to concatenation). If you reserve enough memory beforehand, and if you do not create unnecessary copies of Strings, you will be fine.

I have that hypothesis too, the problem is about how concatenation works. If that really creates intermediary objects, then it’s unavoidable (or not a deal if it gets destroyed after concatenating).

J-M-L:
I have not tested but If you do a

s = String(“IP(“) + WiFi.localIP().toString() + “) is how you find me”;

even If s has plenty of pre-allocated space you will create a temporary hole Since the IP buffer resulting from the function call will only be cleared at the end of your encompassing function. So once executed you will have a 16 bytes hole and the IP String buffer (16 bytes).

But… what if I pre-allocate the main String and then do the concatenation that way?
My current knowledge says that if we start from a clean heap, all temporary objects should be allocated just after the main (bigger) one. When those get destroyed, their occupied spaces also get freed up; but since they were allocated just after the main one, no fragmentation should happen.

I don’t believe the dynamic memory manager will place new objects in random positions when the free space is already contiguous (i. e. just after a restart/reset/reboot).

J-M-L:
“IP(192.168.0.122” would overflow the first temporary String buffer allocation (the one for “IP(“) and thus a new 32 bytes buffer would be reallocated after the IP address String buffer leaving a 16 byte hole that cannot be used for concatenating the “) is how you find me”which will be put in its own extra 32 byte dynamically buffer (which might not be available)

Again, if we start from a clean heap or contiguous free space; fragmentation still shouldn’t occur because all temporary objects and their content (whenever reallocation took place or not) disappear; and in the end is like nothing happened at all.

Robin2:
you won’t be certain even after carrying out experiments.

Makes sense…

demkat1:
Use of 20 Strings, with a length of 10-20 each, and 500more bytes in variables…I dont think that they can create a problem in any way.

Global and pre-allocated, they shoudn’t. Problems occur on the fly, not at the very beginning (sometimes even setup() passes with no hassle).

demkat1:
Except, if user codes

String string21=string1+string2......+string20;

Dont blame “Strings” in such case…

Well… that’s too much and obvious. Even on many programming enviroments for PC things like that are discouraged; since this can trigger the garbage collector more often (unless you use a mutable version to concatenate).

wildbill:
you decide to add some new features for once perhaps not involving new Strings. Then your system starts acting up and you assume that the new features are the cause and try to debug them. But it turns out that the real cause is that you consumed memory that the heap was using for Strings and now fragmentation has occurred. Strings not only bit you, but made it harder to debug.

I can tell for sure this is typical on AVRs (maybe except the 2560), but on other architectures with more RAM perhaps occurs less often or even not at all.

J-M-L:
then when you perform your TestString="";
[…]
→ so the memory stay occupied

Good to know. So that means I can quickly “empty” out a String without losing it’s already allocated memory.

Up until a year or less, I used Strings quite frequently; therefore, I will be the last to berate one who does. The ESP8266 and the 2560 had enough memory to dissolve me of my inadequacies. It wasn't until I combined a project involving a ProMini I began to encounter problems.

For those wishing to wash themselves of the String class, I found the following very easy to implement:

const char *mystring = "..."; const int; const unsigned int; const byte; strncpy - Copy string strcat - Concatenate strings strtol - Convert string to long integer memset - Fill/clear block of memory memcpy - Copy block of memory strtok - Split string into tokens

As described the String for the IP address is not released until the end of the encompassing function - so depending on what you do after - say allocate a longer lasting object that was small enough to fit t within the 16 bytes hole but not completely (say 2 bytes) then you have leaked 14 bytes that the String Class will never use because it allocates in 16 bytes chunk.

Again - if you know what you do then you can be on the safe side - but sometimes things are hidden in libraries you don’t study and they come byte you…

Long story short if you can’t control a clean last allocated first freed type of allocation pattern, you are possibly at risk of poking holes.

J-M-L: ... and they come byte you...

-lol

Lol.. of course this was all intentional and pun intended ...

I used the avr-libc malloc, realloc and free implementations to visualize what the memory looks like.

I tried three different methods of String concatenation.

  • As you can see, directly assigning the result of the ‘+’ operator to the result String will result in memory fragmentation.
  • However, when using a temporary String to save the concatenated result, and then copying its internal buffer to the result variable (after the temporary StringSumHelper buffers etc. are freed), no fragmentation occurs. This is because the resulting String will always use less space than the different parts separately.
  • The same is true when the result String has been pre-allocated: no fragmentation occurs

Code

String returnAsString(const char *cstr) { return {cstr}; }

int main() {
  {
    String res = String("Hello, ") + "World" + '!';
    printf("Done: \"%s\"\n", res.c_str());
  }

  cout << endl;

  {
    String res;
    {
      String temp = String("Hello, ") + "World" + '!';
      res = temp;
    }
    printf("Done: \"%s\"\n", res.c_str());
  }

  cout << endl;

  {
    String reserverd;
    reserverd.reserve(32);
    reserverd = "Hello, " + returnAsString("World") + '!';
    printf("Done: \"%s\"\n", reserverd.c_str());
  }
}

Interpretation

[color=red]X[/color] represents 8 bytes of used heap memory (including 8 bytes of metadata per block)
[color=green]·[/color] represents 8 bytes of free heap memory
[color=orange]R[/color] represents the 8 bytes of metadata at the beginning of a free block of heap memory

Results

Construct String 0 with cstring "Hello, ":
Reallocate buffer for String 0 "(null)":
[color=red]X[/color][color=red]X[/color]
Construct String 1 with String 0 "Hello, ":
Assign String 0 "Hello, " to String 1 "(null)":
Reallocate buffer for String 1 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
StringSumHelper 1
Concat String 1 "Hello, " + "World"
Reallocate buffer for String 1 "Hello, ":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Concat String 1 "Hello, World" + "!"
Reallocate buffer for String 1 "Hello, World":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Construct String 2 with String 1 "Hello, World!":
Assign String 1 "Hello, World!" to String 2 "(null)":
Reallocate buffer for String 2 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 1 "Hello, World!":
[color=red]X[/color][color=red]X[/color][color=orange]R[/color][color=green]·[/color][color=green]·[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 0 "Hello, ":
[color=orange]R[/color][color=green]·[/color][color=green]·[/color][color=green]·[/color][color=green]·[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Done: "Hello, World!"
Destructed and freed String 2 "Hello, World!":


Construct String 3 with cstring "":
Reallocate buffer for String 3 "(null)":
[color=red]X[/color][color=red]X[/color]
Construct String 4 with cstring "Hello, ":
Reallocate buffer for String 4 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Construct String 5 with String 4 "Hello, ":
Assign String 4 "Hello, " to String 5 "(null)":
Reallocate buffer for String 5 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
StringSumHelper 5
Concat String 5 "Hello, " + "World"
Reallocate buffer for String 5 "Hello, ":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Concat String 5 "Hello, World" + "!"
Reallocate buffer for String 5 "Hello, World":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Construct String 6 with String 5 "Hello, World!":
Assign String 5 "Hello, World!" to String 6 "(null)":
Reallocate buffer for String 6 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 5 "Hello, World!":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=orange]R[/color][color=green]·[/color][color=green]·[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 4 "Hello, ":
[color=red]X[/color][color=red]X[/color][color=orange]R[/color][color=green]·[/color][color=green]·[/color][color=green]·[/color][color=green]·[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Assign String 6 "Hello, World!" to String 3 "":
Reallocate buffer for String 3 "":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=orange]R[/color][color=green]·[/color][color=green]·[/color][color=green]·[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 6 "Hello, World!":
[color=red]X[/color][color=red]X[/color][color=red]X[/color]
Done: "Hello, World!"
Destructed and freed String 3 "Hello, World!":


Construct String 7 with cstring "":
Reallocate buffer for String 7 "(null)":
[color=red]X[/color][color=red]X[/color]
Reallocate buffer for String 7 "":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Construct String 8 with cstring "World":
Reallocate buffer for String 8 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Construct String 9 with cstring "Hello, ":
Reallocate buffer for String 9 "(null)":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
StringSumHelper 9
Concat String 9 "Hello, " + "World"
Reallocate buffer for String 9 "Hello, ":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Concat String 9 "Hello, World" + "!"
Reallocate buffer for String 9 "Hello, World":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Assign String 9 "Hello, World!" to String 7 "":
Destructed and freed String 9 "Hello, World!":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Destructed and freed String 8 "World":
[color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color][color=red]X[/color]
Done: "Hello, World!"
Destructed and freed String 7 "Hello, World!":

My complete source code can be found on GitHub: ArduinoStringExperiments

Pieter

JMeller:
For those wishing to wash themselves of the String class, I found the following very easy to implement:
[…]
strtok - Split string into tokens

Man, the rest are easy to use (at least for any somewhat experienced Arduino/C++ programmer); however… strtok and strchr? Those are more or less the toughest to deal; not because they are almost impossible to use, but because they are tricky to use correctly.

J-M-L:
say allocate a longer lasting object that was small enough to fit t within the 16 bytes hole but not completely (say 2 bytes) then you have leaked 14 bytes that the String Class will never use because it allocates in 16 bytes chunk.

Once again, assuming a clean heap, the following String shouldn’t cause fragmentation because you can’t create a new object during the concatenation process (outside of an interrupt routine of course).
The slack will be created only if the free space was fragmented already.

I could be wrong if dymamic memory manager still places new objects in random locations even when free space is contiguous; or if the String’s buffer reallocation method forces it to be in a multiple of 16 position (I’m talking about position, not size).

J-M-L:
but sometimes things are hidden in libraries you don’t study and they come byte you…

Then it’s fault of an unoptimized library, or the silly programmer who doesn’t care about warnings of that library.

By the way, nice pun. :slight_smile: Considering there is also the “nibble”, which is half a “byte”. Who invented those terms anyway? :o

J-M-L:
Long story short if you can’t control a clean last allocated first freed type of allocation pattern, you are possibly at risk of poking holes.

And unfortunately, it’s quite difficult to predict. I don’t even know what’s under the hood of the ESP core libraries I need to use; to be more specific:

  • Wi-Fi controller (ESP8266WiFi).
  • TCP server (web server).
  • UDP client (NTP).
  • Hardware UART controller (aka Serial, hopefully requiring only 642+22 bytes of static allocation).
  • A SHA-256 digester (I know it needs 64 bytes of RAM + some hash tables stored in program memory).
  • Virtual EEPROM or SPIFFS controller.
  • And finally an ESP SDK’s function which is restart() (trigger a reset via Bluetooth terminal/console).

PieterP:
I tried three different methods of String concatenation.

  • As you can see, directly assigning the result of the ‘+’ operator to the result String will result in memory fragmentation.
  • However, when using a temporary String to save the concatenated result, and then copying its internal buffer to the result variable (after the temporary StringSumHelper buffers etc. are freed), no fragmentation occurs. This is because the resulting String will always use less space than the different parts separately.
  • The same is true when the result String has been pre-allocated: no fragmentation occurs

In a few words, am I safe to use String? I hope even if I create substrings locally (necessary to split the command and it’s arguments for the Bluetooth terminal; and extract the sent password from a HTTP POST request), the temporary strings get discarded the same way as in a concatenation.

I could be wrong if dymamic memory manager still places new objects in random locations even when free space is contiguous; or if the String’s buffer reallocation method forces it to be in a multiple of 16 position (I’m talking about position, not size).

One can use this post to figure out what is happening inside the String class and where the actual text is stored; it requires a modification of the String class (WString.h and WString.cpp).

With regards to strtok/strchr for parsing, the advantage is that you do not necessarily have to create new Strings / char arrays to hold the data, you can use the returned pointers to point to the individual parts, even afterwards.

Below a modified version of the tutorialpoints strtok example that will remember the individual parts

void setup()
{
  Serial.begin(57600);

  char *parts[10];
  int counter = 0;


  char str[80] = "This is - www.tutorialspoint.com - website";
  const char s[2] = "-";
  char *token;

  /* get the first token */
  token = strtok(str, s);

  /* walk through other tokens */
  while ( token != NULL )
  {
    parts[counter++] = token;
    token = strtok(NULL, s);

    if(counter >= 10)
    {
      break;
    }
  }

  for (int cnt = 0; cnt < counter; cnt++)
  {
    Serial.println(parts[cnt]);
  }
}

void loop()
{
}