Getting confused with strings

[peter-bos]

Hi,

I'm getting confused working with arrays and Strings. I use Arduino on a ESP8266, so I need to be careful using memory. I want to make a string (I use the word without capitol for both arrays and Strings) in a way that it doesn't corrupt or overrun my memory.

I have a float VOLTAGE, which is formatted as 0.00f, an integer IDX and a boolean VALUE. They should be used in a string like this:
"idx=IDX&value=VALUE&voltage=VOLTAGE"

The length of the string is not constant, since the numbers can have different lengts (like IDX=9 or IDX=10).

How do I concatenate parameters and the fixed text to a string - as said without memory issues?

Peter

[UKHeliBob]

Please post a complete sketch that illustrates what you want to do

[drmpf]

I want to make a string (I use the word without capitol for both arrays and Strings) in a way that it doesn't corrupt or overrun my memory.

My SafeString library was designed to do just that. Check out the detailed tutorial and example sketches that come with the lirbrary, available from Arduino library manager

There is also a forum topic for SafeString usage
https://forum.arduino.cc/index.php?topic=723906

"idx=IDX&value=VALUE&voltage=VOLTAGE"

createSafeString(sfStr,80); // should be long enough but in any case SafeString will prevent overflows

// in setup
SafeString::setOutput(Serial); // turn on debug msg

// in loop
sfStr = F("idx="); // can use F( ) to add char data
sfStr += IDX;  // converts to chars
sfStr += "&value=";
sfStr += VALUE?"true":"false";
sfStr += "&voltage=";
sfStr.print(VOLTAGE,3); // add float to 3 dec.
// could have used print instead of += as well
// you can use sfStr.c_str() to get the underlying char[] or just
// Serial.println(sfStr);

Finally if you want to keep using Arduino Strings (with cap S) follow the guidelines in my tutorial
Taming Arduino Strings

[guix]

You could also "tame" null-terminated char arrays and their functions, they are not so bad... Yes they can be dangerous, if you don't know what you are doing.

With a char array, you can do what you want with sprintf (easiest solution, one line of code) or with strcat, itoa and dtostrf (longer solution but less memory-hungry than sprintf).

Here an example PKz1A7 - Online C++ Compiler & Debugging Tool - Ideone.com

[drmpf]

Here is an example post of someone who thought they know what they were doing
**Use of string char causes Arduino code to restart **(corrected url)

Using raw char[] and low-level c-string methods is very fragile. Small changes in the data can cause the whole sketch to just fail/continually reboot.

SafeString and Arduino Strings, avoid that possibility and so are much safer to program with. SafeString gives you detailed error messages and just keeps going.

c-string methods like strcpy are so bad, Microsoft has banned their programmers from using them and text books have been written on why they should not be used.

[Deva_Rishi]

I use Arduino on a ESP8266, so I need to be careful using memory.

What are you using ? The Arduino or the ESP ?

(I use the word without capitol for both arrays and Strings)

I use 'String' for String and 'string' or 'c-string' for char* or char arrays.

"idx=IDX&value=VALUE&voltage=VOLTAGE"

Looks like you are creating a URL for either a POST or a GET request.
Using c-strings you will have to declare the maximum size of a char array (+1 for the null terminator) , but you can easily declare something that is a fair bit bigger. The idea is to make sure that you don't need to have this memory reserved all the way through your program, and let it go out of scope the moment you are done with it.
Concatenating c-strings can be a bit tricky, but not risky as long as you make sure that a) you don't write beyond the size of the array, and b) always have a Null-terminator at the end.
Using Strings you can declare an object with a variable size in a different part of your memory, and this is where fragmentation may occur. Keep in mind, fragmentation only occurs if you increase the size of different 'Strings' alternatingly. If you create a String, add more String parts to that, and then let the whole thing out of scope, you will not have fragmented anything in the end. Strings can however take up more memory than is required during the concatenation process, hence the question about which board are you actually using ? If you are using an Arduino to control an ESP using AT-commands, you may be looking for the boundary anyway, but then you are using a smaller board (memory and processing speed wise) to control a bigger board, and that is anyway 'Not the way to go'
SafeStrings is another option, it is as simple as that.

[anon73444976]

drmpf:
Here is an example post of someone who thought they know what they were doing
Use of string char causes Arduino code to restart

Here is an example of someone who can't form a URL correctly.

Here is the correct link

[drmpf]

Thanks for that correction

[peter-bos]

Thanks for all the answers!

Deva_Rishi:
What are you using ? The Arduino or the ESP ?

I use the Arduino IDE to write a program for the ESP.

Deva_Rishi:
Looks like you are creating a URL for either a POST or a GET request.

Not the complete URL. The ESP will communicate with another ESP using ESP-Now. The second ESP is via a serial link connected to a third ESP that actually creates a real URL to communicate with Domoticz on a Raspberry Pi. There are two reasons for this setup. First, the ESP-Now connection is faster and less battery consuming than using only one ESP that talks directly with the RPi. Second, this way the ESP's outside the house only have the SSID in the software but not my network password. The stupid thing is that I have no real problem creating this setup, but I got confused on using strings (written with a capitol or not). Maybe that's just because I'm used to Lazarus/Pascal where you hardly have to think about string issues.

Most applications like this that I found on the internet use a record to pack the data from the sensor, with the sensortype as one of those fields. On the receiver side you have to extract the fields from the record and translate this data to a JSON string based on the sensortype. My approach was to send the variable part of this JSON as a string/String to the receiver:

param=switchlight&idx=84&value=0.1&battery=3.21

The receiver completes the URL with the network address and port number of Domoticz and with the fixed part of the JSON:

http://192.168.1.162:8080/json.htm?type=command&param=switchlight&idx=84&value=0.1&battery=3.21

The receiver doesn't need to translate the record which has one big advantage: you don't have to change the receiver software when a new sensor type is added.

Deva_Rishi:
Using Strings you can declare an object with a variable size in a different part of your memory, and this is where fragmentation may occur. Keep in mind, fragmentation only occurs if you increase the size of different 'Strings' alternatingly. If you create a String, add more String parts to that, and then let the whole thing out of scope, you will not have fragmented anything in the end.

Two questions. First, how to let the whole thing go out of scope? Second, is it a solution to create the String with a length which is big enough. Something like:

String theJsonPart = "012345678890123456789012345678890123456789";

for a string of 40 characters and the use it in my program to create a meaningful string:

theJsonPart = "idx="; // the string now has 4 characters, followed by 36 non-used memory bytes
theJsonPart += IDX;
theJsonPart += "&value=";
theJsonPart += VALUE;

and so on.

Peter

[peter-bos]

After reading the text on the page drmpf and TheMemberFormerlyKnownAsAWOL recommended I was wondering if I see to many bears on my path (like we say in Holland). I'll have two types of sensor modules. The first is battery powered and will be in deep sleep between sending messages. When waking from deep sleep the ESP8266 doen't continue where it stopped when it fell asleep, but restarts. Since I only send one message before it goes back to deep sleep, there will be no string/memory issue.
The second type is powered with a USB power supply. Since this sensor only sends the realtime values and doesn't need to maintain a history, I can have that one reset itself every day or couple of days. This also prevents memory issue.

Or do I miss something?

Peter

[drmpf]

Spot on, reset/restarts will solve all your possible String memory problems.
See guideline 8 for ESP32/ESP8266 in Taming Arduino Strings
So just use Arduino Strings. I am preparing a tutorial on handling Serial input/parsing using Arduino Strings.
Drop me a line if you want the examples I have so far.
(Having said that SafeString provides some very convenient Serial read/parsing methods)

Re recovering String (and other) memory

void someMethod() {
   String str;  // small length
   String str1; // another string
   . . . 
   str += "12345";  // reallocates str and leaves hole   
   str1 += "abc";  // leaves another hole
   .. ..
   str += "78910";  // reallocates str and leaves yet another hole in heap memory
}  // all str str1 heap and all stack used recovered here

at the end of the method ALL the str, str1 heap memory is recovered and ALL the holes disappear.
so as long as you use Strings and avoid malloc/calloc and strdup
which create pointers to heap memory that need to be freed later
All the heap and stack memory is recovered when the method exits/returns

[Deva_Rishi]

All the heap and stack memory is recovered when the method exits/returns

The only pitfall left is the use of globally declared 'Strings' and increasing their size while inside a function that uses the heap. (concatenating Strings may also declare a new String, basically if using a 'global' String, use reserve() to declare it's maximum size.

[peter-bos]

drmpf:
Spot on, reset/restarts will solve all your possible String memory problems.
See guideline 8 for ESP32/ESP8266 in Taming Arduino Strings

That guideline was what triggered me.
Peter

[drmpf]

basically if using a 'global' String, use reserve() to declare it's maximum size.

My Taming Arduino Strings tutorial includes a StringReserveCheck class that you can use to check that you have indeed reserved the String's maximum size. There are examples in the tutorial.