Unable to remove the "Reallocating large string" message

Projects Programming
1

Hello,

I am programming an IoT device that uses ESP8266 microcontroller.

The thing is that in the code I am building a message that is sent to a MQTT broker. This message's size is variable and it depends on the number of sensors connected to the device.

I am actually using dynamic memory allocation:

      String *data = new String(buildMessageFromModbusData(associatedDevices, numberOfConnectedNodes, currentUnixTime));

And I am also using an SD card to retrieve unsent values, so I append this SD's message to the one just requested to the sensors:

      if (!data->isEmpty())
      {
        String *sdData = new String();
        if (sdStorage && storedData)
        {
          Serial1.println(F("Retrieving stored data from SD card..."));
          *sdData = SDCardManager::readFile();
          if (!sdData->isEmpty())
          {
            *data = *sdData + "," + *data; // sdData and data sent together as one JSON
          }

At the end of the code, I am destroying the object using delete function.

After all this, my output is:

Retrieving stored data from SD card...

[String] '{"id":1,"t ... 29,0.36000': Reallocating large String(143 -> 144 bytes)

[String] '{"id":1,"t ... ,39.139999': Reallocating large String(159 -> 160 bytes)

[String] '{"id":1,"t ... 999939]} ,': Reallocating large String(165 -> 256 bytes)

Message: {"id":1,"time":1700105400,"data":[229.3999939,0,0,0,0,1,50,60.99000168,0.029999999,61.02000046,1.470000029,39.5,40.97000122,1.470000029,0.360000014,0,39.13999939]} ,{"id":1,"time":1700131960,"data":[225,0,0,0,0,1,50,61.02999878,0.029999999,61.05999756,41]}

So apparently I have some error I am not being able to spot. Can you help me out?

Thanks!

3

The best advice I can give is don't use Strings, use zero terminated arrays of chars (aka C style strings) instead

Declare any such arrays at their maximum size then leave them alone apart from populating them with data

4

Understand. So I should be trying to use some methods such as reserve() instead?

The only problem with that is that the message size is unknown, so I will be oversizing the string memory allocation so it fits.

There is no real problem with this string reallocation. It might cause the program to run slower but at the moment it is not significant. It is more of a visual debugging problem.

5

Yeah I have already seen a lot of people recommending not to use Strings in Arduino. The only problem with that is what I answered to Delta_G: I will have to define the size of the string previously, what is a problem if the message's size is unknown.

7

Using the reserve() function does what its name implies and reserves memory for a String (note the uppercase S)

That is not what I am suggesting. What I am suggesting is using C type strings (lowercase s), which are different altogether

@Delta_G has already answered that

8

What type of data does this function call return?

9

It returns a String, and

SDCardManager::readFile()

returns a String too.

10

What I understand from your message is that I should never be reserving memory for a variable whose size is unknown as it may run an exception if the size is bigger right?

11

So why do you need to duplicate that String?

12

Understand. I am trying this way now. Thanks!

13

Not duplicating. My code tries to retrieve the data from the sensors every 10s and send it to a server. If this data is not sent successfully (no internet connection or connection with the server), I save the value in my SD card.

When I have values in the SD card, what I do when I get my connection back is sent the "instant" value (the every 10s value) plus one of the saved SD card values.

14

Answering you again, I have tried this approach and it removed the 'reallocating large String' message!

Just for anyone with this problem, this is my solution:

String data = buildMessageFromModbusData(associatedDevices, numberOfConnectedNodes, currentUnixTime);
const char *dataCStr = data.c_str();
if (dataCStr != nullptr && strlen(dataCStr) > 0)
      {
        const char *sdDataCStr = nullptr;
        if (sdStorage && storedData)
        {
          Serial1.println(F("Retrieving stored data from SD card..."));
          String sdData = SDCardManager::readFile();
          sdDataCStr = sdData.c_str();

          if (sdDataCStr != nullptr && strlen(sdDataCStr) > 0)
          {
            // Concatenate strings
            char *combinedData = new char[sdData.length() + data.length() + 2]; // +2 for the comma and null terminator
            strcpy(combinedData, sdDataCStr);
            strcat(combinedData, ",");
            strcat(combinedData, dataCStr);

            data = String(combinedData); // Convert back to Arduino String

as you can see, I haven't really stopped working with Strings, but all the conversion is made using C type strings as @UKHeliBob suggested. I have no doubt my code can be optimized, but as a first approach it is more than enough as the problem is fixed!

16

Please post the buildMessageFromModbusData() or better still, post the whole sketch

17

Ok I understand now, thanks!

18 
String buildMessageFromModbusData(NodeModbus nodes[], uint8_t numberNodes, time_t unixTime)
{
  String json;

  for (uint8_t i = 0; i < numberNodes; i++)
  {
    float dataBuffer[nodes[i].device.paramMeasure];

    if (GetDataFromAnalyzer(nodes[i], dataBuffer))
    {
      DeviceData deviceData;
      deviceData.id = nodes[i].address;
      deviceData.time = unixTime;
      deviceData.data = dataBuffer;
      deviceData.arraySize = nodes[i].device.paramMeasure;

      if (!json.isEmpty())
      {
        json += ",";
      }

      json += deviceData.toJson();
    }
  }

  return json;
}

here GetDataFromAnalyzer basically returns the values from the sensor and DeviceData:

struct DeviceData
{
  byte id;              /**< ID of the device */
  unsigned long time;   /**< Time of the data measurement (Unix timestamp) */
  float *data;          /**< Modbus Data (array of float values) */
  int arraySize;        /**< Number of elements in the data array */

  // Method to converts the DeviceData object to a JSON string
  String toJson() const
  {
    DynamicJsonDocument doc(1024);

    doc["id"] = id;
    doc["time"] = time;

    JsonArray dataArray = doc.createNestedArray("data");
    for (int i = 0; i < arraySize; i++)
    {
      dataArray.add(data[i]); // Guardamos el valor con una precision de 3 decimales
    }

    String json = "";
    serializeJson(doc, json);

    return json;
  }
};
19

you are

if your buildMessageFromModbusData() returns a String then this duplicates that String

String *data = new String(buildMessageFromModbusData(•••));

so you need twice the memory than needed at a point in time.

we would still recommend you declare a global buffer of the right max size

const size_t jsonMaxSize = 100;
char jsonBuffer[jsonMaxSize]; // +1 for the trailing null char

your buildMessageFromModbusData() would take a couple more parameters, a pointer to the buffer and the buffer size. This was your function is generic and it's up to the caller to own and manage the buffer.

void buildMessageFromModbusData(NodeModbus nodes[], uint8_t numberNodes, time_t unixTime, char * json, const size_t maxSize)

of course your DeviceData class should also be able to deal with cStrings

20

Hello,

Thanks for your answer. The only problem with that solution is that my buffer has a fixed size, what can be a problem at some point, let me explain:

My sensor is an electrical measurement device, whose number of values is fixed but not the number itself. If values increase over the lifetime of the device (for example happens with energy) I might be exceeding the max size I am defining by software, right?

21

You typically define the buffer large enough for the maximum use case your code needs to handle. If your counter is a uint32_t then the largest number is 4 billions so you need 10 bytes for that part of the message. (Don’t forget also the trailing null char).

This way you don’t have bad runtime surprises, the memory is pre allocated for you. The String class will typically not do what you ask if memory is low but it won’t tell you…

22

Is it a good practice to pre allocate memory in this type of situations then? I will study it and try to achieve a solution.

Thank you so much!

23

If the size of the buffer is going to be a problem at some point if you pre allocate it then it will be a problem when you allocate it dynamically