[ESP32 arduino] Making a HTTP Put request is very slow

I'm trying to upload a 40MB csv file to a cloud server, hence I'm using a PUT request.

void send_file() {
  if ((wifiMulti.run() == WL_CONNECTED)) {
    File file = SD.open("/file0.csv");
    if (!file) {
      Serial.println("Failed to open file");
      return;
    }
    const auto filesize = file.size();

    HTTPClient http;
    http.addHeader("Content-Type", "text/csv");
    http.addHeader("Content-Length", String(filesize, DEC));
    http.setAuthorization(username, password);
    char finalurl[256];
    sprintf(finalurl, "%s/%s", apiUrl, file.name());
    //  Serial.println(finalurl);
    http.addHeader("Content-Type", "text/csv");
    http.addHeader("Content-Length", String(filesize, DEC));
    http.setAuthorization(username, password);
    http.begin(finalurl);
    int http_code;
    const auto A = millis();
    http_code = http.sendRequest("PUT", &file, filesize);
    Serial.printf("\n time-keep: %lus", (millis() - A) / 1000);
    http.end();
    file.close();
  }
}

Is there any way of speeding this up because I've tried to upload the same file using CURL and it happens withing 1-2 minutes. The same operation is taking more than 15-20 minutes from the ESP32

that's on a desktop class machine with large memory buffers I suppose.
You should expect a significant difference with a small ESP...

where is your file stored? (external SD card?)

@J-M-L Yes it's stored on external sd card. Will this affect performance?

Well it's definitely not as fast as the SSD in your PC ...

What's on the receiving end ? curl-ing a 40Megs file from my Mac to another Mac over wifi at home is quite fast. how do you get 2 minutes, this is super slow ?

some thoughts:

The latency can come from many places including the quality of your wifi infrastructure and possible errors during transfer (the read buffer from the SD is likely 512 bytes and transferred in chunks of 1460 bytes, so for a 40 Megs file you are looking at ~82000 SD reads transferred in more than 7000 packets that are each ack-ed and retried if needed)...

The SD card type, its formatting, ... play a role there. the author of SDFat had published some information here Which SD library to use ?!? in an old post. The HTTPClient might depend on the std SD library to read in the stream, so probably not as optimized but quite OK - should be well within a minute to go through the 40 Megs file.

There have been reports and questions in the past on the latency and performance of the socket layer of the ESP32. Their doc states something in the range of 2MBps if I remember correctly but some people reported seeing way less than...

Still this would not explain 15 minutes

In my tests I have never managed to go beyond the upload speed of 120/140 Kb / sec even using the integrated flash memory just to avoid other types of slowdowns.

But I got the best results by using the lowest level WiFiClient / WiFiClientSecure class and splitting the data into predefined packets

Something like that:

define BLOCK_SIZE  TCP_MSS  // Maximum Segment Size (MSS)
.....
  uint16_t pos = 0;
  int n_block = trunc(size / BLOCK_SIZE);
  int lastBytes = size - (n_block*BLOCK_SIZE);

  for( pos = 0; pos < n_block; pos++){
       client.write((const uint8_t *) data + pos*BLOCK_SIZE, BLOCK_SIZE);
       yield();
   }
   client.write((const uint8_t *) data + pos*BLOCK_SIZE, lastBytes);
....

@J-M-L Thank you. The slowness while using curl is likely due to some issues on the server. In my tests I've found out sending a single 40 MB file takes about 25 mins (from the ESP). The internet connection is of 60 MB/sec. Don't know what I'm doing wrong. As @cotestatnt has stated I'll try splitting the data into chunks and using the WiFiClient library.

Thank you @J-M-L and @cotestatnt

When I'm trying to use client.write, I'm getting the error "HTTPClient has no member named write".

This was bugging me so I just ran a quick non optimized test in my local infrastructure (ie Mac and ESP32 on the same Wi-Fi network)

on my Mac I ran in a terminal window:

nc -l 1234 > test.log

which basically listens on port 1234 and dump whatever arrives into test.log until the end of transmission (client goes away).

I ran this code on the ESP32:

#include <WiFi.h>

/* UPDATE THOSE FOR YOUR SYSTEM */
const char* ssid = "****";
const char* password =  "****";
const char * host = "10.0.0.13"; // my Mac running the 'nc -l 1234' command

const uint16_t port = 1234; // the port being used on the Mac

const size_t bufferSize = 1024; // 1kB
uint8_t buffer[bufferSize];

// sends n instances of buffer to the WiFiClient
void blast(WiFiClient& client, size_t n) {
  for (size_t i = 0; i < n; i++)
    if (client.write(buffer, bufferSize) != bufferSize) {
      Serial.println("Data transfer Error");
      break;
    }
}

void setup()
{
  Serial.begin(115200);

  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.write('.');
  }
  Serial.print("\nWiFi connected with IP: ");
  Serial.println(WiFi.localIP());

  for (size_t i = 0; i < bufferSize; i++) buffer[i] = 'a' + (i % 26); // fill buffer with small caps alphabet

  Serial.println("\nrun 'nc -l 1234 > test.log' on your server");
  Serial.println("*** enter key to start ***");
  while (Serial.read() == -1) yield(); // active wait for user to type 'enter'

  WiFiClient client;
  if (client.connect(host, port)) {
    size_t bufferCount[] = {1024ul, 5 * 1024ul, 10 * 1024ul, 20 * 1024ul, 40 * 1024ul}; // 1MB, 5MB, 10MB, 20MB, 40MB
    for (byte i = 0; i < sizeof bufferCount / sizeof bufferCount[0]; i++) {
      uint32_t t0 = millis();
      blast(client, bufferCount[i]);
      uint32_t t1 = millis();
      Serial.print(bufferCount[i]);
      Serial.write('\t'); Serial.print((t1 - t0) / 1000.0, 3);
      Serial.print("s\t"); Serial.print(1000.0 * ((bufferCount[i]*bufferSize) / 1024) / (t1 - t0)); Serial.println(" kB/s");
    }
    client.stop();
  } else {
    Serial.println("Connection to host failed");
  }
}

void loop() {}

the code should be self explanatory, after you hit the enter key in the Serial monitor, I basically send 1MB, 5MB, 10MB, 20MB and 40MB and measure the time it takes and display in the the Serial monitor at 115200 bauds

that's what I got:

.....
WiFi connected with IP: 10.0.0.33

run 'nc -l 1234 > test.log' on your server
*** enter key to start ***
1024	1.822s	562.02 kB/s
5120	8.867s	577.42 kB/s
10240	17.829s	574.35 kB/s
20480	35.328s	579.71 kB/s
40960	64.567s	634.38 kB/s

if I check on the server, test.log does holed 79691776 bytes of 'abcdefghi...xyz'
I send (40960 + 20480 + 10240 + 5120 + 1024) = 77 824 buffers of 1024 bytes and 77 824 x 1024 = 79691776 ➜ all was received correctly.

So I was able to blast from SRAM to an open socket a 40 meg file, in chunks of 1k, at 634.38 kB/s and it took a bit more than a minute from my ESP32. (tried a few times and got up to ~700kB/s)

Reading the buffer from an SD card or flash memory of course would add to this.

HTTPClient library was build on top of WiFiClient but there arent't methods to write directly the stream.

I'm afraid if you want to try this method you have to "create" the request from scratch (without the helps of HTTPClient library).

Great tests @J-M-L

I've missed to mention that my "low" speed measurements was related to uploads to remote server like Google Drive or Telegram using a multipart/form-data request as required from that services.

ah so you have the latency of all the intermediary infrastructure too.... hard to control....

So after a little digging I've found the server to be the issue.

Test A:
Made local server for testing PUT requests. Made 6 Files of 2, 5, 10, 15, 20 and 25 Mb, and put them on the SD Card. Now did the same thing I was trying to do with cloud server, instead this time it was a local server. (Thanks to @J-M-L for the test code). I got the following results.

file1.csv	8.953s	    2mb	
file2.csv	22.093s	    5mb	
file3.csv	43.814s	   10mb	
file4.csv	64.660s	   15mb	
file5.csv	89.897s	   20mb	
file6.csv	104.536s   25mb

Test B:
Same thing as above, but using the cloud server (a WebDAV HTTPS link). Got the following result.

file1.csv	88.989s	   2mb
file2.csv	200.632s   5mb

This could be the server and/or the network latency / bandwidth allocated to your server access throttling

Most probably.