Optimizing String usage - print()

Hello,

Recently I've been working on the usage of String in my ESP8266 Firmware code as I had memory reallocation and inefficient usage of the resources. I have been reading carefully the following guide from drmpf:

At some point, he says:

What is a really simple way of fixing the way Strings are built. However, I was wondering about the print function. Does it affect the fact of using the operator '+' to print a large String? Is this creating short lived temporary Strings?

Thanks,
Diego

The advice that you will get here will most likely be not to use Strings in the first place, but to use C style strings, ie zero terminated arrays of chars

2 Likes

The author is also on this forum. Let's get his attention: Hello @drmpf are you still on this forum ?

1 Like

Yes. The advice to avoid adding Strings is good.

The advice to avoid Strings altogether is better.

No matter you continue to use Strings or not, learning how to do without is essential.

a7

I actually don't want to adapt all my code to use SafeString, but there are some interesting notes in their documentation about the usage of String:

Definitely a debate.

He actually says they are not that bad if you use them correctly:

Arduino Strings are not as bad as they have been made to be and by following a few guidelines, can be used with confidence in your sketches. Arduino Strings use minimal extra SRAM memory over a plain char[], about 8bytes/String, but that 8 bytes provides you with complete safety from buffer overruns and missing string terminators as well as convenient coding methods.

If you want to keep complete control over your memory usage or you are working with a library that returns a char* or you want a richer set of readers, parsers and text functions or want detailed debugging/error messages, then you can install the SafeString library from the Arduino library manager.

Either of these alternatives, String or SafeString, are preferable to using error prone low level char[] manipulations and c-string methods, like strcat and strcpy, which are the systemic cause of so many programming errors.

I've got my :popcorn: ready

2 Likes

Yes, a debate, opinions and stuff.

Please learn how to use C/C++ character arrays. Whether or not you continue to use Strings or SafeStrings or whatever.

That's all I am saying. I never use Strings because I grew up when there was no such thing, and I started with tiny microprocessors where using anything like that would have been ridiculously ridiculed as well as not working very well.

a7

1 Like

the SafeString library is written using the "dangerous" functions the authors does not like - yet he relied on those to write his library... (let's not start that debate again !)

that's because c-strings are not bad if you use them correctly.

Knowing to deal with buffers (arrays) and ensuring you are staying within the bounds of your buffer is a critical notion to master, be it for strings or general purpose buffers. So that's something any wannabe developper should master

Once you have that skill, you can decide what you want to use

For prints using String + the following works well

 {
 String str = String ... + .... 
 Serial.println(str);
 }

Any memeory allocated is completely recovered at the closing } without any holes in the memeory

But being ESP you still need to restart regularly to clean up the memory.
Either from the ESP's own String use or from memory leaks in its support code.

see Taming Arduino Strings Avoiding Fragementation and Out-of-Memory Issues

Also see the code in GitHub - boblemaire/IoTaWatt at 02_06_03
for how to monitor the free memeory and reboot as necessary.
(not sure exactly where in there, but I can see the logs saying low memory restart)

I didn't want to refute you, I am actually in a process of learning and the forum and the users' opinions are extremely valuable for me.

However, I don't always take someone's opinion as the absolute truth, and I have seen a lot of controversy on this issue. That's why I wanted to have the debate.

Thank you so much for the information, I will study C char arrays deeply :stuck_out_tongue:

Understood, I've noticed that many of you share the same opinion. I will study them!

Thank you so much for your response.

Just as you say, I have seen a bunch of conflicting opinions.

When it comes to something as code developing, I had never seen this kind of controversy, so I wanted to have a justified answer to why are things done like this. Even to why String exists when is not even necessary.

instead of concatenating Strings before printing, just print the parts one by one.
Or even just stream to your printable hardware.

It depends on what you want to achieve.

What does your sketch look like today?

Thanks for coming!

I will take a look to the links, even though from this post's responses I think I will get to study C char arrays and try to stop using Strings.

Well I guess I understood it now.

I'm in the process of optimizing my code and the different opinions on this topic had confused me. Everthing is clear now, thank you so much :smiley:

You certainly need to understand char arrays (and arrays in general)
But avoid the low level c-string routines unless you are a perfect programmer who never makes mistakes.
Strings were introduces over 20 years ago in response to the persistant coding errors in using c-strings.
Those who do not study history are doomed to make the same mistakes.

1 Like

In a nutshell there is one key notion in this debate which is your appetite for understanding memory management for collections of objects and the associated trade offs

You can go with "static, fixed size" or "dynamic variable size" memory allocation.

Strings are not the only buffer you’ll need (think about binary protocols or just collections of values in an array, vectors, matrix, graphs, and any other type of data set), so for those of us with enough programming experience and willingness to understand what’s under the hood the answer is clear : you need to know how to handle the static fixed size case as it’s a pattern you’ll see a lot in C or C++.

Once you master that pattern you’ll see there are many functions to handle the fixed size constraint and you can just code with those or decide you are tired of working at that level and then you write your own set of helper functions or classes to encapsulate the frequent operations. You do this by building on the knowledge you acquired and carefully address the potential edge cases and overflow issues. This is what also led to more classes in C++ like the containers.

This encapsulation comes at a cost (a few more bytes, a few more computing cycles, possible non universal API so non portable or not optimized knowledge).

When you work on small microcontrollers sometimes every byte and every cycle matters and if you are a diligent and careful programmer then you can save a bit here and there to fit into the hardware constraints.

In some cases understanding what’s happening in memory is a choice between life and death. It’s no joke. So for those of us who have been exposed to such responsibility or just care about not crashing, you can understand there is a strong position on staying in control with proven functions.

But the crux is there: The native functions or any library built on top of static usage won’t handle edge cases for you. The good ones just give you a way to catch the issue. How you handle that is for the programmer to solve and lazy / bad / tired coders just don’t code for it, thinking "it will never happen in a normal use case and too bad for you if you did not follow what’s expected". That’s not counting on hackers looking for this... :imp:

Amongst the edge cases, buffer overflow is one of the most common issue developers face and need to code for. With price of memory going down and more capable hardware around, some thought that the best way to solve for this edge case was by not having the programmer worry about: if the buffer is too small, just make it bigger.

And that works until you hit memory limits…

On modern 64 bits computers with an OS, Gigs of RAM and virtual memory this limit is quite far and you only get a speed impact if you have to swap a lot.

On small microcontrollers with no OS and where you play with actual RAM and no virtualization it’s another story.

One question then is whether the library you use catches the exception that there is no memory available for buffer expansion and how this is presented to the developer to handle. (try/catch)

One challenge you have with the Arduino String class is that this is left uncaught, the operation fails silently and can either crash or just not do anything leading the programmer to believe all went fine.

(The other challenge is that your code could actually have been fine with a larger static buffer but rules for dynamic allocations lead to a situation where there was no block of memory large enough to accommodate the expansion - that’s the challenge of poking holes into the heap and having no garbage collector)

➜ if you use the String class you can’t catch the memory limit issue and thus your code is at risk, you are no longer in control.

It all boils down to Is it a risk you can live with?

2 Likes

So as I understand:

Using c-strings requires a deeper understanding of the language, but when used effectively, they represent the most efficient approach. This is specially crucial when we talk about low memory microprocessors and it needs to be as optimized as possible.

String were introduced by Arduino 20 years ago as a response to avoid using c-strings. They are not that "optimized" but simpler to use. They can be used in more advanced projects but following some use directives in order to do not break the program.

In this forum most of the people are against the usage of String, what is justified by multiple reasons.

Wow thank you so much for such a well explained response. I think I have really understood the use case scenario about all this and where the real problem resides.

In my particular project, I don't currently face memory, hacking, or 'life or death' issues, so perhaps the change isn't truly worth it at the moment. But I'm sure I'll consider it for future projects.

Above all, I've realized that c-strings are a milestone in my dev career.