"null" at the end of string from file.readStringUntil()

[blackhoody]

Im working with esp32s3 feather right now. I need to log some data when there is no WiFi connection. Write works fine for me but when I want to read line with readStringUntil(), i always get "null" at the end of read string. Here is code:

if ((millis() - sdLast) > sdTime)
{
  for (int i = 0; i < maxSensors; i++)
  {
    if (activeSensors[i] != "")
    {
      String requestData = "{\"data\":[{\"name\":\"" + sensorNames[i] + "\" ,\"temp\": \"" + actTemp[i] + "\",\"hum\": \"" + actHum[i] + "\",\"time\": \"" + actTime[i] + "\",\"scanCount\": \"" + scanCount[i] + "\"}]}\n";
      appendFile(SD, "/all.txt", requestData.c_str());
      sdReady = true;
    }
  }
  sdLast = millis();
}

Function to read from file:

void readLinesSD(fs::FS &fs, const char *path)
{
  File file = fs.open(path);
  WiFiClient client;
  HTTPClient http;
  http.begin(client, serverName);
  http.addHeader("Content-Type", "application/json");
  if (!file)
  {
    Serial.println("Failed to open file for reading");

    return;
  }
  while (file.available())
  {
    buffer = file.readStringUntil('\n');
    serializeJson(doc, buffer);
    Serial.println(buffer);
    int httpResponseCode = http.POST(buffer);
    Serial.println(httpResponseCode);
    doc.clear();
    delay(200);
  }
  http.end();
  file.close();
}

Append function:

void appendFile(fs::FS &fs, const char *path, const char *message)
{
  Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if (!file)
  {
    Serial.println("Failed to open file for appending");
    return;
  }
  if (file.print(message))
  {
    Serial.println("Message appended");
  }
  else
  {
    Serial.println("Append failed");
  }
  file.close();
}

SO basically I want to save data to file and then, when the WiFi connection is back I want to send data to database for further presentation. When I read file i got this results:

{"data":[{"name":"P RHT 902631" ,"temp": "19.53","hum": "48","time": "1674746950","scanCount": "4"}]}null
{"data":[{"name":"P RHT 90262A" ,"temp": "19.38","hum": "50","time": "1674746957","scanCount": "4"}]}null
{"data":[{"name":"P RHT 902629" ,"temp": "19.36","hum": "49","time": "1674746958","scanCount": "5"}]}null

I tried using some special characters like "%" at the end of lines and then read line untill this special character but got same problem. When I used the same function on my other esp32 board everything was read fine. Anyone know what might cause this problem? Thanks for any help

[UKHeliBob]

  buffer = file.readStringUntil('\n');

What is the data type of the buffer variable ?

[blackhoody]

buffer is String type variable

I also tried using other special characters instead of \n, like % or & and read string untill them but got same thing, null at the end

[J-M-L]

How is your memory usage ? With so many Strings (and hidden ones through using +) you might run out of memory and some of the operations just fail silently

Side note
In readLinesSD() you begin your http transaction but might exit (return) the function without closing it

[blackhoody]

How can I check memory usage? Sorry if it some trivial question but have no idea xd

[J-M-L]

It’s not a trivial question because the way stack size is allocated and heap managed.

You can Read docs such as

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/mem_alloc.html

https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/heap_debug.html#heap-information

For testing purpose though I would suggest you just try minimizing what you write may be using just a static text to get started

Instead of

String requestData = "{\"data\":[{\"name\":\"" + sensorNames[i] + "\" ,\"temp\": \"" + actTemp[i] + "\",\"hum\": \"" + actHum[i] + "\",\"time\": \"" + actTime[i] + "\",\"scanCount\": \"" + scanCount[i] + "\"}]}\n";
appendFile(SD, "/all.txt", requestData.c_str());

Do

appendFile(SD, "/all.txt", "{\"Hello\"}\n");

And see if you still have the null (also can you clarify where you get that ? Is it the arduino serial monitor or on the other HTTP side?)

[blackhoody]

I will try apeending shorter text to a file. I printed read string with Serial.println() and when I wanted to send it to my server, it send it with null at the end, os got http error aswell

EDIT:
Well, my mistake. Serial.println() after reading the file dont give me null at the end.

 while (file.available())
    {
        buffer = file.readStringUntil('\n');
        Serial.println(buffer);

        http.begin(client, serverName);
        http.addHeader("Content-Type", "application/json");
        serializeJson(doc, buffer);
        Serial.println(buffer);
        int httpResponseCode = http.POST(buffer);
        Serial.println(httpResponseCode);
        doc.clear();
        delay(500);
    }

Instead, Serial.println() after serializeJson(doc,buffer) shows in terminal nulls at the end

[J-M-L]

How is doc defined?
Try catching the error on Serialization
(Look at the size you obtain and compare with buffer size)

Actually on second thought what are you trying to do? The json is in buffer, don’t you want to call deserializeJson instead ??
What’s the doc purpose?

[blackhoody]

What Im trying to do is to send data to my server, data which is collected when there is no WiFi connection or power is off, so I want to save it to file and then send it to database after Wifi is back.

I thought i could do something like in loop, there I do POST every 60 seconds on server

        if (WiFi.status() == WL_CONNECTED)
        {
            JsonArray array = doc.createNestedArray("data");
            for (int i = 0; i < maxSensors; i++)
            {
                if (activeSensors[i] != "")
                {
                    JsonObject object = array.createNestedObject();
                    object["name"] = activeSensors[i];
                    object["temperature"] = actTemp[i];
                    object["humidity"] = actHum[i];
                    object["time"] = actTime[i];
                    object["scanCount"] = scanCount[i];
                }
            }
            WiFiClient client;
            HTTPClient http;
            String requestBody;
            http.begin(client, serverName);
            http.addHeader("Content-Type", "application/json");
            serializeJson(doc, requestBody);
            int httpResponseCode = http.POST(requestBody);
            Serial.println(httpResponseCode);
            doc.clear();
            http.end();
        }

[blackhoody]

I guess problem is solved for now at least, I messed a bit. Without serializeJson(), http.POST() works and dont get any nulls or http errors.

[J-M-L]

@blackhoody

Without serializeJson

my view was that there was no point calling it in the first place

[blackhoody]

Yeah, I just realised that it is totaly unnecessary here, thanks for all your help

[J-M-L]

(and I think that's what was messing up with your String)