Sending large Data from esp32 to Server

Using Arduino Sensors
1

Greetings!

I am sending data from the esp32 to a server which I first record from the sensors(4 data points per minute) to the spiffs for an interval like 30-60 minutes and then turn on the wifi, read the data from the spiff and send it to the server using HTTP requests.

The problem is when the data is 60 minutes, it works fine and transfers all the data after 60 minutes, but when I exceed it from 60 minutes it does not send the data to the server while the data is recorded correctly and printed on serial monitor.

What could be the possible reason? Is it because the data is too large?

Kindly assist

2

can you please provide your sketch?

3

How many bytes are you saving in SPIFFS ?
Which partition scheme have you got selected in the IDE ?

4

I am using this function to send data.

void sendDataToServer(const String & payload) {
  // Print the payload for debugging
  Serial.println("Payload to be sent to server:");
  Serial.println(payload);
  HTTPClient http;
  http.begin(serverName);
  http.addHeader("Content-Type", "application/json");

  int httpResponseCode = http.POST(payload);

  esp_task_wdt_reset(); // Reset the watchdog timer

  if (httpResponseCode > 0) {
    Serial.print("HTTP Response code: ");
    Serial.println(httpResponseCode);

    String response = http.getString();
    Serial.println(response);
  } else {
    Serial.print("Error code: ");
    Serial.println(httpResponseCode);
  }

  http.end();
}

And the payload contains lines like this:

file.printf("{\"macAddress\":\"%s\",\"IPAddress\":\"%s\",\"timestamp\":\"%s\",\"temp\":%.2f,\"accel_x\":%.2f,\"accel_y\":%.2f,\"accel_z\":%.2f,\"battery_voltage\":%.2f,\"battery_percentage\":%.2f,\"state\":\"%s\"}\n",
      data.macAddress.c_str(), data.ipAddress.c_str(), data.timestamp.c_str(), data.temperature, data.accelerationX, data.accelerationY, data.accelerationZ, data.batteryVoltage, data.batteryPercentage, data.state.c_str());
5

I have used a 1.2 MB spiff partition. Also, I am monitoring the spiffs usage in my code and it is a lot, hardly 2 hours of my data will be about 40kbs.

7

I read data from the spiff to the payload like this:

Serial.println("Reading data from SPIFFS:");
    String payload = "["; // Start JSON array
    while (file.available()) {
      String line = file.readStringUntil('\n');
      if (line.length() > 0) {
        if (payload.length() > 1) {
          payload += ",";
        }
        payload += line;
      }
    }
    payload += "]"; // End JSON array
    file.close();
8

If I were you, I'd upload the whole sketch in one piece, so someone can try use it on their own to understand the problem better, and it just makes it a little easier to understand exactly what's going on.

9

Are you in control of the server as far as the code ?
What response code are you getting back from the server i.e. 200 ?
Can you confirm number of sent bytes by the esp32 (client) ?

10

Yes, I am controlling the server side too and everything is perfect on that side.
I get 400 when I exceed the interval from 1 hour.
How may I know how many bytes are being sent? I can only see it once they are on the server.

11

Count how many you put into SPIFFS

12

Oh, that I can see 52710 bytes used in spiff.

13

What does that mean? No request to the server at all? You're not checking the return value for http.begin. What's the httpResponseCode?

String has a max size

#ifdef BOARD_HAS_PSRAM
        enum { CAPACITY_MAX = 3145728 }; 
#else
        enum { CAPACITY_MAX = 65535 }; 
#endif

Does your board have PSRAM? If it does, you may have to enable it, in the Tools menu, as one of the board-specific options listed below Get Board Info.

If it should fit, you might also try to reserve the space in the String in advance; the file size plus some extra for the added brackets and commas.

1 Like
14

You are printing out payload so you should be able to see what is being sent to the server. You could also use payload.length()

You said 4 data points per hour so why is spiffs have 52k worth of data ? May try littleFS instead of spiffs.

Also, 400 response from server means it did not like the request ie. it is not correctly formatted. I would double check your print out of the payload and see if it is valid json.

There is also a service called postman that will let you send request to your server. I don't know if you are familiar with that program but it will help you sort out what is going on. It is worth learning how it works.

I second the request to put up all your code.

15

It's 4 data points per minute.
It works fine when I send less than one hour of data, so it is formatted correctly.

And yeah I checked it on Postman, copied the data of 3 hours from the serial monitor and sent it through Postman, it worked well, so no error on the server side too.
Now I'm sending it in chunks again, why can't I send it in one request?

16

Maybe you should monitor what comes over the serial port, maybe your stack overflows and your device resets.