Go Down

Topic: Webserver sending files to client (Read 1 time) previous topic - next topic

dario_show

Hello everybody.
I'm working on a simple webserver using Arduino MKR1010 WIFI.
My sketch is based on the "WifiWebServerSketch" that you can find in the examples of WiFiNINA library.
I analize the GET request received from the client and I sent him the response.
If the client requests a file present in the SD card, then I use a function (written by myself) that uses a buffer to read the file from the SD card and send it to the client using the client.write(buf, len) function, in order to speed-up the transmission (as known, reading and sending 1 single byte at a time is a waste of time).
I use the number of bytes written from the instruction client.write to check if the transmission has been succesfully completed.
My problem is this: sometimes the client can receive the complete file and sometimes the file transmission stops randomly at some point (in this case I can even try to send again the content of my buffer but it doesn't help and the instruction client.write keeps returning zero bytes transmitted, so I abort the transmission).
I tryed this for small (some KBytes) or "big" (some MBytes) files but I get the same problem.
Am I forgetting something? Is there somebody that got the same problem in similar application?

Juraj

do you set content-length header?
esp8266 has client.write(file). ESP32 doesn't have it?

dario_show

do you set content-length header?
esp8266 has client.write(file). ESP32 doesn't have it?
No, up to now I send always the same header without sending the content-length field. I will try to send the correct header and tell if it works.

Juraj

No, up to now I send always the same header without sending the content-length field. I will try to send the correct header and tell if it works.
my WebServer

dario_show

Adding the content-length header (like in your code) didn't solved completely the problem, even if now seems better.
I still got the transmission stopped randomly in some cases during these last days.
Using Chrome tools I can see that when it stops, I get the error "ERR_CONTENT_LENGTH_MISMATCH 200 (OK)" for my GET request and the write instruction returns less byte than expected.
Anyway I'm still learning how HTTP transmission works and I need to find some good guide for how to create a complete and correct header from arduino.
But it sounds strange to me that most of the times the transfer completes without errors or problems and sometimes stops   :smiley-roll:
Can you suggest me an example sketch using data transfer with buffer?

Juraj


dario_show

Ok, I will need some time to cut a lot of things and make the sketch minimal so it will be easier to understand and debug.
But what happen and what should I do when the "client.write(buf, len)" returns less bytes than "len"?

Juraj

Ok, I will need some time to cut a lot of things and make the sketch minimal so it will be easier to understand and debug.
But what happen and what should I do when the "client.write(buf, len)" returns less bytes than "len"?
process len bytes and then read again. or use readBytes. it will wait for bytes

dario_show

I try to send only the part of code where I get the problem, I think the complete code is too long and not very neat for now. This function takes as argument a file path and send it to the client. It uses an array as a buffer.
For example if SD_BUFFER_SIZE = 10 and client.write sends only 7 bytes, then I wait for 5ms and then try again to transmit the last 3 bytes from the buffer.
Changing SD_BUFFER_SIZE or the delay time doesn't help, 95% of the times the file transfer completes
 without problems, the other times it stops in a random point. I would like to achieve a more reliable comunication.
P.S. up to now I can download files from Arduino at about 100 kbyte/s


#define SD_BUFFER_SIZE 1460
char fileBuffer[SD_BUFFER_SIZE];

void sendFileFromSD(String filePath) {
  File fileToSend = SD.open(filePath, FILE_READ);
  unsigned long FileSize, fileSendStartTime, totalByteSent;
  int totalRetryCount = 0;
  if (fileToSend) {
    FileSize = fileToSend.size();
    fileSendStartTime = millis();
    // BEGIN OF HEADER
    client.println(F("HTTP/1.1 200 OK"));
    client.print(F("Content-Length: "));
    client.println(FileSize);
    client.println(F("Connection: close"));
    client.print(F("Content-Type: "));
    switch (clientRequestType) {
      case FILE_HTM:
        client.println(F("text/html"));
        break;
      case FILE_TXT:
        client.println(F("text/plain"));
        break;
      case FILE_JPG:
        client.println(F("image/jpeg"));
        break;
      case FILE_PNG:
        client.println(F("image/png"));
        break;
      case FILE_JAVASCRIPT:
        client.println(F("text/javascript"));
        break;
      default:
        client.println(F("text/plain"));
    }
    client.println(); // empty line to close header
    // END OF HEADER
    totalByteSent = 0;
    boolean readError = false, writeError = false;
    while ((totalByteSent < FileSize) && (!readError) && (!writeError)) {
      int bufferLength = SD_BUFFER_SIZE; // init with buffer max size
      if (bufferLength > FileSize - totalByteSent)
        bufferLength = FileSize - totalByteSent;
      int partialByteRead = fileToSend.read(fileBuffer, bufferLength);
      if (partialByteRead < bufferLength) {
        readError = true;
      }
      else {
        int SendCount = 0; // first send attempt
        int bufferByteSent = 0;
        while ((bufferByteSent < bufferLength) && (!writeError)) { // until all reads byte are not sent ...
          int remainingByteToSend = bufferLength - bufferByteSent;
          int partialByteSent = client.write(&fileBuffer[bufferByteSent], remainingByteToSend);
          bufferByteSent += partialByteSent;
          totalByteSent += partialByteSent;
          if (partialByteSent < remainingByteToSend)
            delay(5); // wait some time if transfer has not been completed
          SendCount++;
          if (SendCount > 20) // maximum 20 times then abort
            writeError = true;
          if (SendCount > 1)
            totalRetryCount++;
        }
      }
    }
    fileToSend.close();
  }
}

dario_show

I begin thinking nobody is using Arduino for file transfer.
I wish Arduino documentation could be better and could explain what to do in case the transfer stops. Maybe is only a bug?
Even the function client.flush() is not clear to me: when should it be called?

Juraj

I begin thinking nobody is using Arduino for file transfer.
I wish Arduino documentation could be better and could explain what to do in case the transfer stops. Maybe is only a bug?
Even the function client.flush() is not clear to me: when should it be called?
https://github.com/jandrassy/StreamLib/blob/master/README.md

and see how simple is the file serving in my WebServer with BufferedPrint

dario_show

Thank-you for your help, Juraj.
Sure, I'm not expert but I'm trying to write my code only using the WiFiNINA library, hoping that all its functions are well tested.
Anyway, I opened the source code of the library and found an interesting thing: the code of the flush() function is this:

void WiFiClient::flush() {
  // TODO: a real check to ensure transmission has been completed
}


So it seems this function is doing nothing, not as documentation says.
This makes me think the authors didn't complete this part (and maybe some others??) and I can't rely on this library  :'(

Juraj

Thank-you for your help, Juraj.
Sure, I'm not expert but I'm trying to write my code only using the WiFiNINA library, hoping that all its functions are well tested.
Anyway, I opened the source code of the library and found an interesting thing: the code of the flush() function is this:

void WiFiClient::flush() {
  // TODO: a real check to ensure transmission has been completed
}


So it seems this function is doing nothing, not as documentation says.
This makes me think the authors didn't complete this part (and maybe some others??) and I can't rely on this library  :'(
it is ok. they don't do buffering so there is nothing to flush.
use BufferedPrint for buffering

Go Up