Webserver sending files to client

Hello everybody.
I'm working on a simple webserver using Arduino MKR1010 WIFI.
My sketch is based on the "WifiWebServerSketch" that you can find in the examples of WiFiNINA library.
I analize the GET request received from the client and I sent him the response.
If the client requests a file present in the SD card, then I use a function (written by myself) that uses a buffer to read the file from the SD card and send it to the client using the client.write(buf, len) function, in order to speed-up the transmission (as known, reading and sending 1 single byte at a time is a waste of time).
I use the number of bytes written from the instruction client.write to check if the transmission has been succesfully completed.
My problem is this: sometimes the client can receive the complete file and sometimes the file transmission stops randomly at some point (in this case I can even try to send again the content of my buffer but it doesn't help and the instruction client.write keeps returning zero bytes transmitted, so I abort the transmission).
I tryed this for small (some KBytes) or "big" (some MBytes) files but I get the same problem.
Am I forgetting something? Is there somebody that got the same problem in similar application?

do you set content-length header?
esp8266 has client.write(file). ESP32 doesn't have it?

Juraj:
do you set content-length header?
esp8266 has client.write(file). ESP32 doesn't have it?

No, up to now I send always the same header without sending the content-length field. I will try to send the correct header and tell if it works.

dario_show:
No, up to now I send always the same header without sending the content-length field. I will try to send the correct header and tell if it works.

my WebServer

Adding the content-length header (like in your code) didn't solved completely the problem, even if now seems better.
I still got the transmission stopped randomly in some cases during these last days.
Using Chrome tools I can see that when it stops, I get the error "ERR_CONTENT_LENGTH_MISMATCH 200 (OK)" for my GET request and the write instruction returns less byte than expected.
Anyway I'm still learning how HTTP transmission works and I need to find some good guide for how to create a complete and correct header from arduino.
But it sounds strange to me that most of the times the transfer completes without errors or problems and sometimes stops :roll_eyes:
Can you suggest me an example sketch using data transfer with buffer?

show your sketch

Ok, I will need some time to cut a lot of things and make the sketch minimal so it will be easier to understand and debug.
But what happen and what should I do when the "client.write(buf, len)" returns less bytes than "len"?

dario_show:
Ok, I will need some time to cut a lot of things and make the sketch minimal so it will be easier to understand and debug.
But what happen and what should I do when the "client.write(buf, len)" returns less bytes than "len"?

process len bytes and then read again. or use readBytes. it will wait for bytes

I try to send only the part of code where I get the problem, I think the complete code is too long and not very neat for now. This function takes as argument a file path and send it to the client. It uses an array as a buffer.
For example if SD_BUFFER_SIZE = 10 and client.write sends only 7 bytes, then I wait for 5ms and then try again to transmit the last 3 bytes from the buffer.
Changing SD_BUFFER_SIZE or the delay time doesn't help, 95% of the times the file transfer completes
without problems, the other times it stops in a random point. I would like to achieve a more reliable comunication.
P.S. up to now I can download files from Arduino at about 100 kbyte/s

#define SD_BUFFER_SIZE 1460
char fileBuffer[SD_BUFFER_SIZE];

void sendFileFromSD(String filePath) {
File fileToSend = SD.open(filePath, FILE_READ);
unsigned long FileSize, fileSendStartTime, totalByteSent;
int totalRetryCount = 0;
if (fileToSend) {
FileSize = fileToSend.size();
fileSendStartTime = millis();
// BEGIN OF HEADER
client.println(F("HTTP/1.1 200 OK"));
client.print(F("Content-Length: "));
client.println(FileSize);
client.println(F("Connection: close"));
client.print(F("Content-Type: "));
switch (clientRequestType) {
case FILE_HTM:
client.println(F("text/html"));
break;
case FILE_TXT:
client.println(F("text/plain"));
break;
case FILE_JPG:
client.println(F("image/jpeg"));
break;
case FILE_PNG:
client.println(F("image/png"));
break;
case FILE_JAVASCRIPT:
client.println(F("text/javascript"));
break;
default:
client.println(F("text/plain"));
}
client.println(); // empty line to close header
// END OF HEADER
totalByteSent = 0;
boolean readError = false, writeError = false;
while ((totalByteSent < FileSize) && (!readError) && (!writeError)) {
int bufferLength = SD_BUFFER_SIZE; // init with buffer max size
if (bufferLength > FileSize - totalByteSent)
bufferLength = FileSize - totalByteSent;
int partialByteRead = fileToSend.read(fileBuffer, bufferLength);
if (partialByteRead < bufferLength) {
readError = true;
}
else {
int SendCount = 0; // first send attempt
int bufferByteSent = 0;
while ((bufferByteSent < bufferLength) && (!writeError)) { // until all reads byte are not sent ...
int remainingByteToSend = bufferLength - bufferByteSent;
int partialByteSent = client.write(&fileBuffer[bufferByteSent], remainingByteToSend);
bufferByteSent += partialByteSent;
totalByteSent += partialByteSent;
if (partialByteSent < remainingByteToSend)
delay(5); // wait some time if transfer has not been completed
SendCount++;
if (SendCount > 20) // maximum 20 times then abort
writeError = true;
if (SendCount > 1)
totalRetryCount++;
}
}
}
fileToSend.close();
}
}

I begin thinking nobody is using Arduino for file transfer.
I wish Arduino documentation could be better and could explain what to do in case the transfer stops. Maybe is only a bug?
Even the function client.flush() is not clear to me: when should it be called?

dario_show:
I begin thinking nobody is using Arduino for file transfer.
I wish Arduino documentation could be better and could explain what to do in case the transfer stops. Maybe is only a bug?
Even the function client.flush() is not clear to me: when should it be called?

and see how simple is the file serving in my WebServer with BufferedPrint

Thank-you for your help, Juraj.
Sure, I'm not expert but I'm trying to write my code only using the WiFiNINA library, hoping that all its functions are well tested.
Anyway, I opened the source code of the library and found an interesting thing: the code of the flush() function is this:

void WiFiClient::flush() {

  • // TODO: a real check to ensure transmission has been completed*
    }

So it seems this function is doing nothing, not as documentation says.
This makes me think the authors didn't complete this part (and maybe some others??) and I can't rely on this library :cry:

dario_show:
Thank-you for your help, Juraj.
Sure, I'm not expert but I'm trying to write my code only using the WiFiNINA library, hoping that all its functions are well tested.
Anyway, I opened the source code of the library and found an interesting thing: the code of the flush() function is this:

void WiFiClient::flush() {

  • // TODO: a real check to ensure transmission has been completed*
    }

So it seems this function is doing nothing, not as documentation says.
This makes me think the authors didn't complete this part (and maybe some others??) and I can't rely on this library :cry:

it is ok. they don't do buffering so there is nothing to flush.
use BufferedPrint for buffering