Esp32 image transfer not all ftp tcpip packets arrive

Hi, I have a script that runs on a ESP32-CAM that takes pictures and transfer these pictures to a FTP server from time to time.

I notice that when my WiFi network is slow, mostly due to many people using at the same time, images tend to arrive missing a strip at the bottom.
The size of the missing strip varies but usually, at least the first say, quarter, of the image usually arrives ok.
Image size is VGA (around 25K);

The missing strip is just blank but the image size is on the FTP side matches the ESP32-CAM image size verified on the Serial Console and there is no error either on the FTP server or the ESP32.
To consider the image transfer successful, immediately after the loop that transfer the image (below), I check ftp.isConnected(), if true I understand the image was successfully transferred, but that is not the case, it is true both for the image fully transferred but also when a strip is missing.

The code fragment for the FTP transfer is as below:

  imgSize      = ImgMetaData.imSize;
  numLoops     = (ImgMetaData.imSize / bufferSize) + 1;
  for(counter=1; counter<=numLoops; counter++){
    stuffPixels((unsigned char*)tempImageBuffer, startIndex, 0, bufferSize);
    startIndex += bufferSize;    

I tried buffer size 256 and also 1024 and there is no change.
Also tried delayTimer() for 0.1, 0.25 but no difference.

Any idea of how to deal with this problem?


Not working with exactly this kind of transmission I ask You if there is any way of getting an acknowledge that the packet is received by the router and the channel, the router is ready for the next package?

"the router is ready for the next package?"
Don't know how I can check this.

That is the only manifestation of the slow WiFi, all PCs and smartphones deal with the slow WiFi without missing anything.

I suppose You have tried, and succeeded, when the communication load is low on the router.
I can think of trying using smaller packets, let's say half the size.
Sometimes small SMSs were passing but longer messages just disappeared in the GSM system in the old days. That looks like Your situation now.

Send a CRC value along with the data, and check it at the receiving end.

Tried packets 1500, 1024 and even 256 bytes, same behavior.
In fact I do not know if increasing the delay between bursts (as in the for loop on the code above) will result in distinct packets. Have no way to check it.

I understand that, as it works fine when there is no load on the network ( I don't even know if the problem is in my WiFi network or just the consequence of overuse on my branch of the ISP net) and it is sure not at the FTP server I, again, assume packets are been dropped somewhere on their way to the FTP server and the TCPIP implementation on the ESP32 is just not able to track packets acknowledged by the FTP server so the ESP32 then is just not aware packets have been dropped and must be retransmitted.
May be someone with specific knowledge on how TCPIP is implemented on the ESP could give me a clue if there is a way out of this.

"Send a CRC value along with the data, and check it at the receiving end."

Packets that arrive show up on the image, packets that fail to arrive don't.

No, when you receive a bad CRC, request a re-transmission. You make your own CRC, so you don't need any knowledge of TCP/IP to do it. The CRC is on the payload, there may be CRC on the packets, but that isn't relevant to this...

"No, when you receive a bad CRC, request a re-transmission."
That is when a bad CRC is received but what if packets are just dropped due to excess traffic somewhere?
The ESP32 should have a way to detect a timeout when acknowledgement of packet received by the FTP Server do not arrive back and would send the same packet again but it does not seem to be working this way.

And also, I am assuming packets are been dropped, I do not know if this is the case.

You said the packages arrive. Perhaps they are just forwarded with incomplete data after timing out. If you can receive a packet, you can check the CRC. It doesn't matter if there were time outs or not. It also doesn't matter whether the incomplete images are due to transmission errors or time outs. The CRC will catch both situations.

Probably, there is nothing you can do to improve the network performance. That is dictated by the configuration and loading of the network. But you can at least filter images by rejecting bad ones and re-sending them.

For me to resend the picture the ESP32CAM has to know the picture arrived damaged.
The FTP server has no mechanism to tell the ESP32 of this problem as the FTP server is detecting no error. Sending CRC will not help as the FTP server will not be able to report back to the ESP.

Why not? There is nothing in FTP to confirm a complete transmission?

I know of no way to make the FTP interact back with the C++ script in my ESP.

C++ doesn't have scripts. Do you have a link to the FTP library?

I know of no way to make the FTP server to inform my c++ script running on my ESP32-CAM that the file transfer failed.
Do you know of any way to do that?

Link to the library?

Have no idea how to do it.

Where did you get the FTP functionality, then? Is it built into the ESP32 API?

Yes, but it is on the ESP side, the ESP side see no errors, the FTP runs on a hosting service, all I can do is transfer files to it. There is no error anywhere in the process so the problem takes place but both the FTP and the ESP are quite happy that the process completed with no errors.
On the ESP side all I can check is if the connection with the FTP is still UP at the end of the file transmission and it is.
It does not help to, somehow, detect the error on the FTP side if I know of no way to communicating it back to the ESP for file retransmission.