I have discovered that the WiFiClient.write() function does not really send the data supplied to the other end of the connection. Instead it seems to wait for more data in order to be more efficient, I assume. If I send small chunks of data with 1 second intervals they are sent as a packet per second even though they arrive at the ESP8266 through a serial line at fairly low baud rates, like 4 bytes per milliseconds max. Seems like these get collected during a pretty long time interval before finally being sent.
But that will break our application for a serial<=>tcp bridge use.
We use two phases, one sends small command packets of a few bytes only back and forth and expect these to arrive instantly.
Then there is also longer data transfers lasting maybe a minute or so. I have verified that while we send the large chunks they are transferred in big TCP packets and the throughput becomes UART baud rate limited only.
I had a look at available methods for WiFiClient and found flush() but I cannot find any good description of what it does and going to the library file only puts me at a virtual placeholder...
What I am looking for might not be flush(), I need some way to ensure that the data so far put into the transmit buffer by way of write() will actually be stuffed into a TCP packet and sent right away during the negotiation phase.
How can this be done?
Thanks,
I tried it but the packets still contain the bytes I am sending in one single block every 1000 ms.
I am using a 50 byte string which is sent to the device serial line every 1000 ms. It operates at 38400 baud so the UART transfer time is 13 ms. The loop() should execute more often than that, I believe, so I expected that there would be some packet splits now. But no, the packets tick in one every second....
It means that the data are first collected into the TCP buffer and only when no more are written for a certain timeout (anyone knows how long?) the packet is sent. And it contains all data....
When can one know that serial data have been received?
How often is Serial.available() updated?
I have tested now in my loop() function to log information when I vary the amount of data sent over the serial port to the ESP8266 every 1000 ms.
I have added a debug logging at the point where the data gets received, then sent out through the TCP socket:
char serbuf[512]; //Buffer for serial server
//check UART for data, FIFO is 128 in size
while ((bytesAvail = SerialSS.available()) > 0)
{
bytesIn = SerialSS.readBytes(serbuf, min(sizeof(serbuf), bytesAvail));
if (bytesIn > 0)
{
//push UART data to connected telnet client
if (tcpServerClient && tcpServerClient.connected())
{
tcpServerClient.write((uint8_t*)serbuf, bytesIn);
tcpServerClient.flush(); //2018-11-10 BB: Try to force sending
SerialDebug.println("Count: " + String(bytesIn)); //2018-11-10 BB: Show count
delay(0); //Needed to stop wdt resetting on long operations
}
}
}
When the client connects I am also setting the nodelay:
if(tcpServerClient.connected())
tcpServerClient.stop(); //Stop previous connection
tcpServerClient = tcpServer->available(); //Assign new connection to object
tcpServerClient.setNoDelay(true); //2018-11-10 BB: To speed up transmission of data
The result is really surprising!
As I said my external test application is sending a block of characters every 1000 ms.
If I set the block at 20 chars the debug logging shows Count: 20 every second, if I increase to 110 chars it shows Count: 110 every second.
How can this be?
It takes 28 ms to send that amount of data and the loop surely executes more often that that!
So why is the Serial.available() value not updated as data arrive on the UART? It seems like this value is not updated in real time.....
How can I make it supply the correct incoming byte count every time it executes?
The test application is using Windows COM9, which is a USB-to-Serial cable using the FTDI chip and driver.
I have now modified the test application sending the data so I can adjust the number of characters in a block more easily and now I have discovered that if I send 1..138 characters every 1 second I get 1 tcp packet.
But if I increase to 139 chars (or more) in the block then it splits into 2 tcp packets (127 + 12 chars).
It looks like there is some connection to the ESP8266 UART FIFO size even though I don't understand the mechanism...
BUT:
If I reduce the block size AND the interval so I send 4 bytes with 25 ms interval, then the packets reduce to size 4 bytes...
And they are sent at a higher speed too.
Could it be such that the Serial.available() function is detecting a streaming situation (no gaps between bytes) and then waits to supply a value until either the FIFO plus software buffer is full (because I see data sizes above FIFO size 128 bytes) or there is a gap in incoming data?
don't use bytesAvail as a parameter to readBytes. the parameter should be the buffer size. readBytes will wait for additional bytes. at default it will wait a second. it is too much. use SerialSS.setTimeout(20) for example. it will wait 20 milliseconds for more data
Just to clarify:
I thought that Serial.available() returns the number of bytes actually ready to be retrieved at that time.
Then using that value to actually read them seems appropriate to get exactly that amount, since they are reported to be ready. Why would it wait for additional data in this situation?
The issue I am seeing and wondering about is the fact that available() seems to return a value only when the UART is no longer seeing any inbound data. Is this actually how it works?
I will have to figure out a way to send a packet of data and then insert a very short pause (like a ms or so) before sending a new packet.
Then I would see if these are treated as individual packets or as one single packet (depending on the length of the pause, maybe).
Not so easy in Windows, I know...
(I am using Freepascal and Lazarus on Windows 7 to create the test applications)
I will now try the setTimeout() method in the firmware.
serial is slow. there are gaps between data if you see it with the mcu speed. in general it is practical not to read and send the 2 bytes reported as available, but to wait for the next byte some microseconds and then next until nothing is available in timeout or the output buffer is full. that way you fill the output buffer and don't send small chunks.
SerialSS is a define for Serial, SerialDebug is a define for Serial1. I use these to make it clearer when reading the code where the data go since I also have debug output to Serial1.
I have tried to find out if there is a requirement on where I set the timeout, but found no info...
Should this be done before or after Serial.begin?
I am not after optimizing package size to reduce network traffic since in operation there are only two devices on the network, the ESP8266 in AP mode and an Android device connected to the AP.
But I really want to have a good responsiveness so that small exchanges (only a few bytes) do not get delayed.
When there are larger data transfers they are streamed in hardware (no PC or such involved, they come from a microcontroller).
So if I interpret what you suggest, setting the timeout very small, say a few ms will cause the streamed data to be put in full size packets (since new data arrive all the time) while the 3-char commands will be sent off in small packets after the timeout, right?
PS: Thank you for your very useful comments! DS
Forgot to show how I now have arranged the init code:
SerialSS.begin(ESPConf.baud);
SerialSS.setTimeout(5);
SerialDebug.println("Swapping pins for Serial, use 7/10 for Rx/Tx!");
SerialSS.swap(); //Put the serial pins on 7 (Rx) and 10 (Tx) to avoid boot data to reach SS
a quick look into source code suggests that the esp8266 has RX hardware buffer for UART and the core adds a software buffer. available reports sum of both buffers
Juraj:
you could copy every byte from serial to client.
That is what I think I am doing already in the loop()...
a quick look into source code suggests that the esp8266 has RX hardware buffer for UART and the core adds a software buffer. available reports sum of both buffers
How does the timeout affect this?
I understood it as:
When reading the SerialSS.available() byte count it would report what is in the FIFO buffer plus the software buffer (size is set where?) with a timeout of in my case 5 ms. If no data arrived within those 5 ms then the call finishes with a value that can be processed.
When setting the timeout is that to be done before or after Serial.begin? I do it now after as shown above but I see no change in behavior....
OK, thanks.
So:
The copy bytes function does not need a buffer in the loop() and it will finish its job regularly so to allow loop() to terminate and restart?
Or can it be stuck there for a long time triggering a watchdog reset? Since it is basically one single statement without any delay(0) or yield....
Or is the Serial baudrate slow enough to make it run out of data in time? Even while streaming 100K of data...
SerialSS.begin(ESPConf.baud);
SerialSS.setTimeout(2);
//New approach suggested by Juraj on Arduino Forum:
if (tcpServerClient && tcpServerClient.connected())
{
while (SerialSS.available())
{
tcpServerClient.write(SerialSS.read());
}
}
Now I am getting much better response, it actually sends a lot of packets indicating that there are no more extra delays anywhere.
I set my test up for sending 20 bytes at 5 second interval so I could stop it after a single transmission
Result: 10 packets each containing 2 bytes of payload.
But when I change to send 2000 bytes only 604 reach the destination....
I will increase the timeout a bit.
Nope,
Setting it to 10 does not help either.
Not setting the timeout at all does not work either...
If I send blocks of data less than about 280 they seem to transfer fine in small packets, but going above this loses data, not all that are sent get to the target.
with this while loop on esp8266 it reads what Serial at 38400 baud collected in the buffer, while the rest of the loop and WiFi handling was executing.
timeout doesn"t apply for read(). Only for readBytes*, readString*, parse* methods
try to add && client.availableForWrite() to while loop
But where are the bytes lost?
It is depending on the size of the data packet being sent....
In my original code I use readBytes, which then is affected by the timeout setting, right?
But I could not get the smaller packets I was looking for when using that code, which stuffs the data into an intermediate buffer before writing to the socket. OTOH it never lost a single byte...
readBytes seem to have no overload without parameters, like the buffer to stuff data into....
BosseB:
But where are the bytes lost?
It is depending on the size of the data packet being sent....
In my original code I use readBytes, which then is affected by the timeout setting, right?
But I could not get the smaller packets I was looking for when using that code, which stuffs the data into an intermediate buffer before writing to the socket. OTOH it never lost a single byte...
readBytes seem to have no overload without parameters, like the buffer to stuff data into....
in your previous code readBytes wasn't affected by timeout because you constrained it to available bytes.
try while (SerialSS.available() && tcpServerClient.availableForWrite())
serial buffer will hold the bytes until wifi sends the previous batch. then again you copy them fast from serial buffer to WiFi buffer.
I am not posting to get help, just to give it to future questions.
you have the option to use WiFiClient.setSync (true); -> then the writes are safe to be received on the endpoint, as it flush would be. check this: Clase Client — documentación de ESP8266 Arduino Core - 2.4.0 sorry it is in spanish but you can use google translator or deepl.