post request extremely slow

i use esp32 to upload saved to SPIFFS files to dropbox Although i've managed to make it working right, it is VERY slow..i tried to upload a small photo which i had saved in SPIFFS but it takes ages. I simplified my code and the problem is not SPIFFS..it is my POST slow..

here is an example:

client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer *** \r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myTime + ".txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " + chunkSize + "\r\n\r\n");
    Serial.print("chunkSize:");
    Serial.println(chunkSize);
    Serial.print("BEFORE Post time:");
      unsigned long beforeMillis = millis();
    
    Serial.println(beforeMillis);
    
     int counter = 0;
    while (1) {
      client.print(counter);
      client.print("\n");
      counter++;
      if(counter > chunkSize) break;
    }

    Serial.print("AFTER Post, elapsed time:");
    unsigned long afterMillis = millis()- beforeMillis;
    Serial.println(afterMillis);

Example 1 Terminal: chunkSize:1000 BEFORE Post time:3655 AFTER Post, elapsed time:25752

Example 2 Terminal: chunkSize:100 BEFORE Post time:3667 AFTER Post, elapsed time:2415

Example 3 Terminal: chunkSize:10000 BEFORE Post time:3550 AFTER Post, elapsed time:94567

Is this normal? If i have to post a picture of 1mb it will take ages..

Content-Length must be in bytes and must be correct.

use client.write(file);

thanks for both answer.

Content-Length must be in bytes and must be correct.

I use

File myFile = SPIFFS.open("/small.txt");
unsigned long myFileSize = myFile.size();

client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer *** \r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myTime + ".txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " + myFileSize + "\r\n\r\n");

I know that content-length is the size of the BODY only…right? so the code above should be fine…
(i also checked it with this)

  1. do i have to put “Connection: close”?

  2. i have tried client.write(myFileSize) but it looks also slow… i have read that anyway its the optimal solution (it manages the file and send the appropriate chunks), but i cannot find the documentation…

now my code is like this: (hello.txt is 18 bytes)

unsigned long myPostSize;


    //------------------------------WHAT YOU WILL SEND
    bool sendStringNotFile = false;

    if (sendStringNotFile) {
      myPostSize = myString.length();
    }
    else {
      String filename = "hello.txt";
      myPostSize = myFile.size();
    }
    Serial.print("myFileSize: ");
    Serial.println((String)myPostSize);

    // Make a HTTP request:
    client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer *** \r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myTime + ".txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " +  myPostSize + "\r\n" +
                 "Connection: close\r\n\r\n");


    Serial.println("BEFORE Post");
    unsigned long beforeMillis = millis();

    if (sendStringNotFile) {
      Serial.println("i send the string");
      client.print(myString);
    }
    else {
      Serial.println("i send the file");
      while (myFile.available()) {
        char c = myFile.read();
        client.write(c);
      }
    }

    Serial.print("AFTER Post, elapsed time:");
    unsigned long afterMillis = millis() - beforeMillis;
    Serial.println(afterMillis);

terminal (file uploaded fine):
BEFORE Post
i send the file
AFTER Post, elapsed time:43
headers received
dropbox response:
18c
{“name”: blablabla…}
0

replace:

while (myFile.available()) {
     char c = myFile.read();
     client.write(c);
}

with

client.write(myFile);
//the same happens with
//client.print(myFile);
//the same happens with
//client.print(myFile.read());

and i dont get any server answer in terminal:

BEFORE Post
i send the file
AFTER Post, elapsed time:4
(nothing after that)

after tried:

while (myFile.available()) {
          client.write(myFile);
}

instead of just client.write(myFile)…

terminal stuck at:
i send the file
(i guess it is just looping)

i guess the whole spiffs to client is a no go
so i tried read file to string…and worked:

String myFileStr = "";
      while(myFile.available()){
        //doesnt work without char cast
        myFileStr += (char)myFile.read();
      }
      Serial.println("myFileStr:");
      Serial.println(myFile);
      client.print(myFileStr);

now…i went for the big file-SIZE: 456267 (the save code as above)…
myFileStr is printed fine(ok it takes a few seconds).
but i dont receive any server response…

it stuck at:
AFTER Post, elapsed time:19500

no response after…so what? no big files uploading?

cut in chunks the big string didnt work either:

for (int i = 0; i < myFileStr.length(); i = i + 100) {
    client.print(myFileStr.substring(i, i + 100));
}

you use some old version of the esp32 arduino core?

https://github.com/espressif/arduino-esp32/blob/0f772270fbb32099cad72053755d6b4c7e8e1f82/libraries/WiFi/src/WiFiClient.cpp#L413

Juraj: you use some old version of the esp32 arduino core?

https://github.com/espressif/arduino-esp32/blob/0f772270fbb32099cad72053755d6b4c7e8e1f82/libraries/WiFi/src/WiFiClient.cpp#L413

no..im in 1.04..

i have spend more than 10-15 hours..and still a simple POST cannot be done.. the strange thing is that ive seen other examples where they post using string chunks..... but here it does get any response..

PS: i have upload the same "big" files with dropbox API explorer..and it works fine..(same headers) etc..

edit: client(stream) doesnt exist in WifiSecure... it does exist these

uzer123: edit: client(stream) doesnt exist in WifiSecure... it does exist these

class WiFiClientSecure : public WiFiClient

WiFiClientSecure enhances WiFiClient. it inherits the write(stream) function

there is no better way then the client.write(file). if it doesn't work, then there is some error, but it is how it should work.

how do you read the response?

how do you see the inheritance? i expected something like: WifiSecureClient: WifiClient

this is my code:

//WIFIClientSecure

  Serial.println("\nStarting connection to server...");
  if (!client.connect(hostDropbox, 443))
    Serial.println("Connection failed!");
  else {
    Serial.println("Connected to server!");
    String url2 = "/2/files/upload";
    File myFile = SPIFFS.open("/saved.txt");
    String myString = "Hello apo to esp32";

    unsigned long myPostSize;


    //------------------------------WHAT YOU WILL SEND
    bool sendStringNotFile = false;

    if (sendStringNotFile) {
      myPostSize = myString.length();
    }
    else {
      myPostSize = myFile.size();
    }
    Serial.print("myPostSize: ");
    Serial.println((String)myPostSize);

    // Make a HTTP request:
    client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer ***\r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myTime + ".txt\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " +  myPostSize + "\r\n" +
                 "Connection: close\r\n\r\n");

  

    Serial.println("BEFORE Post");
    unsigned long beforeMillis = millis();
    Serial.println(beforeMillis);


    if (sendStringNotFile) {
      Serial.println("i send the string");
      client.print(myString);
    }
    else {
      Serial.println("i send the file");
      Serial.print("time:");
      unsigned long timeMillis = millis() - beforeMillis;
      Serial.println(timeMillis);
      client.write(myFile);
    }
    Serial.print("AFTER Post, elapsed time:");
    unsigned long afterMillis = millis() - beforeMillis;
    Serial.println(afterMillis);

    while (client.connected()) {
      Serial.println("connected o server");
      String line = client.readStringUntil('\n');
      Serial.println(line);
      if (line == "\r") {
        Serial.println("headers received");
        break;
      }
    }

    // if there are incoming bytes available
    // from the server, read them and print them:
    Serial.println("dropbox response:");
    while (client.available()) {
      char c = client.read();
      Serial.write(c);
    }
    client.stop();
  }

the terminal output:

BEFORE Post
4444
i send the file
time:2
AFTER Post, elapsed time:5
connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

connected o server

edit: is there any possibility that because my text file is not urlencoded, i have these problems?

So what is going on in "client.write(myFile);"? Are single bytes being read/sent? Does this function bundle the bytes into a large packet so they are all sent at once?

zoomkat:
So what is going on in “client.write(myFile);”? Are single bytes being read/sent? Does this function bundle the bytes into a large packet so they are all sent at once?

there is a link to it in one of my comments. it uses a 1300 bytes buffer

update: uploading esp32-cam photo works like a charm

//dont use fb->len in Content-Length..you must cast to String
String imgLen = (String)fb->len

client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer ***\r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myTime + ".jpg\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " + imgLen + "\r\n\r\n");
    
    client.write(fb->buf, fb->len);

edit: only in very small pictures (320 x 240)...

is esp32 limited to upload only very small files?

what is fb?

its frame buffer..

camera_fb_t * fb = esp_camera_fb_get();

https://github.com/espressif/esp32-camera

you can adjust the resolution of the frame..but when i change to >SVGA it get stuck..

//FRAMESIZE_UXGA (1600 x 1200) //FRAMESIZE_QVGA (320 x 240) //FRAMESIZE_CIF (352 x 288) //FRAMESIZE_VGA (640 x 480) //FRAMESIZE_SVGA (800 x 600) //FRAMESIZE_XGA (1024 x 768) //FRAMESIZE_SXGA (1280 x 1024)

and SVGA is just 17kB! client.write doesnt need to cut the buffer in chunks..or it does it itself?

edit: maybe it HAS limitation https://github.com/esp8266/Arduino/issues/1872

uzer123: its frame buffer..

camera_fb_t * fb = esp_camera_fb_get();

edit: maybe it HAS limitation https://github.com/esp8266/Arduino/issues/1872

that issue is ancient.

you specify a size in the http request Conten-length header. so write many bytes in the client. no matter in how many writes, but the TCP packets have around 1300 bytes for payload so writing buffers of this size will make fastest transport

actually i finally made this work…after i dont know 50 hours??
i didnt know that the payload have some limitation (is it the same for every client or just esp32? or pc send in packets without me knowning?)

SOO, the problem was that the POST body that was actually sending (you can find it by:

 int howManyBytesSent = client.write(...) )

was SMALLER than the actual body, in BIG files…

So, obviously client.write cannot be used for big files (big can be more than 3-4 kbytes, i dont know the threshold), so as someone suggested here you can cut the post in chunks of 1000 bytes (maybe it can be a bit more), but i had to work with uint8_t * buf so the procedure to make the chunks wanted a little bit of more “magic”…

So my final code is like this (some Serial print checks also included):

int imgLenInt = fb->len;
String imgLenStr = (String)imgLenInt;

client.print(String("POST ") + url2 + " HTTP/1.1\r\n" +
                 "Host: " + hostDropbox + "\r\n" +
                 "Authorization: Bearer ***\r\n" +
                 "Dropbox-API-Arg: {\"path\": \"/" + myFileName + ".jpg\",\"mode\": \"add\",\"autorename\": true,\"mute\": false,\"strict_conflict\": false}\r\n" +
                 "Content-Type: application/octet-stream\r\n" +
                 "Content-Length: " + imgLenStr + "\r\n\r\n");


    int i = 0;
    int myChunkSize = 2000; //if 2000 doesnt work, change to 1000-1500
    char tempCharArr[myChunkSize];
    for (i = 0; i < myPhotoSize - myChunkSize; i = i + myChunkSize) {
      for (int k = 0; k < myChunkSize; k++) {
         char c = myPhoto.read();
        tempCharArr[k] = c;
      }
      howMuch = client.write((const char*)tempCharArr);
      sumHowMuch += howMuch;
    }
    Serial.print("i=");
    Serial.println(i);
    Serial.println("outOfLoop");
    int left = myPhotoSize % myChunkSize;
    Serial.print("left=");
    Serial.println(left);
    howMuch = client.write(myPhoto + (i - myChunkSize), left);
    sumHowMuch = (sumHowMuch - myChunkSize) + howMuch;
    Serial.println("sumHowMuch:");
    Serial.println(sumHowMuch);

ALSO: just to note for esp32-cam i had to set config.jpeg_quality = 15; the default values…dont work properly (the data are uploading but the picture file is destroyed)

now how is it different from what client.write(stream) does?

   size_t toRead = 0, toWrite = 0, written = 0;
    size_t available = stream.available();
    while(available){
        toRead = (available > 1360)?1360:available;
        toWrite = stream.readBytes(buf, toRead);
        written += write(buf, toWrite);
        available = stream.available();
    }

Juraj: there is a link to it in one of my comments. it uses a 1300 bytes buffer

You have a link? The one I saw looks to be for a pi.

Juraj: now how is it different from what client.write(stream) does?

   size_t toRead = 0, toWrite = 0, written = 0;
    size_t available = stream.available();
    while(available){
        toRead = (available > 1360)?1360:available;
        toWrite = stream.readBytes(buf, toRead);
        written += write(buf, toWrite);
        available = stream.available();
    }

lol..now i see the whole client.write function..and yes look really similar.. ok..ill try to keep everything same and exchange only this function to see if there was another problem..

edit: i have the same question..how the number 1360 come up?

uzer123: edit: i have the same question..how the number 1360 come up?

TCP packet payload common size

zoomkat: You have a link? The one I saw looks to be for a pi.

in comment #4

Juraj:
TCP packet payload common size

hmm…here it is networking - Maximum packet size for a TCP connection - Stack Overflow

Also, just to note:

esp32-cam photo, is in the fb->buf datatype (uint8_t *) so you cannot use size_t WiFiClient::write(Stream &stream)

AND client.write(fb->buf, fb->len); doesnt work also…(the packets optimization doesnt seems to happen)…
So my solution is:

int howMuch = 0;
int sumHowMuch = 0;
int i = 0;
int myChunkSize = 1300;
for (i = 0; i < imgLenInt - myChunkSize; i = i + myChunkSize) {
      howMuch = client.write(fb->buf + i, myChunkSize);
      sumHowMuch += howMuch;
}
int left = imgLenInt % myChunkSize;
howMuch = client.write(fb->buf + (i - myChunkSize), left);
sumHowMuch += howMuch;
Serial.println("is your body bytes the same as your sent bytes?");
Serial.print("sent bytes:");
Serial.print(sumHowMuch);

about Files…File is not a stream…so how this can work? do i miss something?

File myPhoto = SPIFFS.open("/cat.jpg");
client.print("....");
int sentSize  = 0;
sentSize = client.write(myPhoto);
Serial.print("you had to send:");
Serial.print(myPhotoSizeStr);
Serial.print(", and you send:");
Serial.println((String)sentSize);

output:
you had to send:19, and you send:1

    int sentSize  = 0;
    int sumSentSize = 0;
    while(myPhoto.available()){
      sentSize = client.write(myPhoto.read());
      sumSentSize += sentSize;
    }
   Serial.print("you had to send:");
    Serial.print(myPhotoSizeStr);
    Serial.print(", and you send:");
    Serial.println((String)sumSentSize);

output:
you had to send:19, and you send:19

but image is corrupted…
small text file uploaded fine…
big text file stuck forever…