Ethernet shield 2 - received only 2048 of 10K bytes

Hello,

I mounted an Ethernet shield 2 on a Due. As API I am using Ethernet library, version 2.0.

In a single write call, I send a buffer of 10K bytes to a remote client over LAN:

#include <Ethernet.h>

std::array<byte, 10000> bufferOut = {};
EthernetServer server(Port);
...
EthernetClient client = server.available();
...
if (client.available()) {
    const std::uint32_t nbWritten = client.write(bufferOut.data(), bufferOut.size());

The number of bytes written returned is 10K. However on the receive side I only get 2048 bytes. The receiver is a simple QT client running on my PC:

#include <QTcpSocket>

socket = new QTcpSocket(this);
        connect(socket, &QIODevice::readyRead, this, [this]() {
        	const std::uint64_t bytesAvailable = socket->bytesAvailable();
        	...
        	const std::uint64_t nbBytesRead =  socket->read(bufferIn.data() + nbBytesReadTotal, bufferIn.size() - nbBytesReadTotal);

The QIODevice::readyRead callback is called one or two times to get a maximum of 2048 bytes in total, though 10K bytes were sent.
Could this be an issue with the 2048 bytes TX socket buffer of the Wiznet W5500 Ethernet chip? Increasing the socket buffer size by reducing the number of sockets in Ethernet.h makes it possible to send larger buffers.
Since TCP is stream orientated, it should be able to transmit any buffer sizes chunking them on its own, independently of the socket buffer size. Did I miss something?
Using a remote QT test server instead of the Arduino server, everything works as expected.

Any help appreciated.
Simon

do you stop the client right after the write?

Juraj:
do you stop the client right after the write?

No, I don't stop the client after sending.

@pjrc

Juraj:
@pjrc

Sorry, what does that mean?

simon_due:
Sorry, what does that mean?

user pjrc is Paul Stoffregen the author of the 2.00 version of the Ethernet library

https://www.pjrc.com/arduino-ethernet-library-2-0-0/

You can open an issue in GitHub repository

Ok, I’m here. I am listening. I’ve subscribed to notifications for this thread.

I do believe this is likely to be a bug in the library. It’s probably not handling large writes (larger than the Wiznet socket buffer) properly. All the code was designed around the concept of memory limited boards like Arduino Uno.

As a possible workaround, try calling availableForWrite() to learn how much data the socket is ready to accept, and then write only that much. Maybe surround that with a loop which keeps retrying until all the data is written. Hopefully that will solve your immediate needs?

Now for the difficult part…

I’m not going to touch this unless you do much more. If you want me to put engineering time into this bug, at the very least I need complete code for both the Arduino side and the PC side. I’m going to be picky about that code. I’m NOT going to mess with Qt. Likewise on the Arduino side, if the code is filled with C++ stdlib & template stuff like “std::array<byte, 10000>”, I’m probably not going to touch it. Yeah, I know that’s arbitrary & a bit harsh, but the reality is I really need to be working on the next Teensy model so I just can’t put a lot of time into Ethernet right now. If I’m going to look into this problem, I need relatively simple and small (but complete) code I can copy into Arduino and run on a board (and I’m not so excited about working with Arduino Due, mostly because of its slow upload speed - so I really want to see a minimal program if I’m going to work with Due). Likewise on the PC side, I really need something small and simple, ideally using only the C standard library stuff with printf() to stdout, which I can compile on Linux and run on the command line to see the result.

Even then, it may be weeks until I actually do anything. I poured nearly near a month of solid time into Ethernet 2.0.0 earlier this year, and to be frank, that’s really used up all the time I can spend on it this year, and then some. Later in 2019 (after the new Teensy board is released), I do want to put more work into Ethernet, mainly for non-blocking APIs and a few commonly requested features. But the reality is all my Arduino contributions are funded by sales of Teensy, and it’s now been 2 years since I’ve released any new hardware! If I don’t keep sustaining a somewhat profitable business, then I’d have to go back to a corporate job and probably never manage to contribute much of anything.

If you want me to look into this issue, please give me small & simple but complete code to reproduce the problem.

pjrc:
All the code was designed around the concept of memory limited boards like Arduino Uno.

Hi Paul,

Good to have your statement. That explains a lot.

Actually I can live pretty well with current limitations. As you proposed, one may quite straightforwardly work around that issue.

I understand very well that you would like to work on a minimal example as basic as possible without high-level language fuss.
To be honest, I am doing this for work. So my capabilities to contribute are rather small. Since there is no immediate need for improvement, I would stick to the current implementation, which is already very helpful.

A non-blocking API would be perfect, maybe I'll need to work on it myself.

A) Paul - If this is still on your to-do list, I hope that your changes can extend into the UDP parts of the library. I recently tried to send a 12012 byte datagram from an MKR Zero/ETH combo to another computer. Only 2048 bytes arrived at the destination. In its (their) defense, the Ethernet library did report that only 2048 bytes were transmitted through the total Ethernet/WizNet 5500 stack.

If you need an example program, I can supply my MKR transmitter code. The receiver is Java running in a PC. I doubt you want that code, but if you do, let me know.

B) Regardless of whether the library changes, are you a person who can ensure that either the ETH card(s) descriptions, or the Ethernet Library description, prominently mention the 2048 byte limit? I didn't spot any mention of that limitation when I was choosing my Hw/Sw stack. I'm able to work around it, but in that regard, other folks might not be as lucky as I am.

Thanks in advance!

Metron_Ross:
are you a person who can ensure that either the ETH card(s) descriptions, or the Ethernet Library description, prominently mention the 2048 byte limit?

No, sorry. Arduino recently started requiring 2 factor authentication for commit access to their repositories. I do not have that set up, and I intentionally do not have a cell phone within reach of my workbench (far too distracting), so until I look into other ways to set that up, I not longer even have access to commit changes on github. I will eventually set up 2FA, but right now with the release of Teensy 4.0 I'm very busy with so many other things (like improving the serial monitor so Arduino doesn't crash when Teensy 4.0 transmit much faster than any other boards).

Even with that access, I have no control over Arduino website. You could try opening an issue on github to request changes. If you do, I highly recommend you give them the exact words and be very specific about the suggested change. You'll also need patience! Usually it takes many months before anyone looks at those sorts of requests, and when they do it often seems to be someone not highly familiar with the tech details, so give then the exact words to copy & paste, and be very specific about exactly where to paste them.

Thank you Paul.