Dealing with RX/TX Serial Buffer Size

Good Day All,

Im hoping that someone can assist me with methods to resolve/manage Serial RX/TX buffers.

The issue is that I am using a ESP8266 WiFi module connected to the Arduino using Hardware Serial1. Things are working fine until I noticed that listing Access points did not return the full list. The arduino only reported the first 1 or 2. I then found that increasing the RX buffer from 64 to 256 solved this issue.

For me this is only temporary, If I was to request the ESP to "GET" a webpage I'm sure it would return more than 256 bytes. How then can I deal with receiving large quantity of data over serial? Is it merely a matter of faster polling? The way I visualize it, is I request a website from the ESP module, it compiles the full response internally then blasts all the data out. Will increased polling really solve the issue as all this data comes in a single blast? Or is their a way that as soon as RX data is being received we tell the micro to focus on getting it all then once all is received continue normal execution?

How then can I deal with receiving large quantity of data over serial?

How do you read a long post? One character at a time. You read serial data the same way. The key is to do that more often than data arrives. Fortunately, that is relatively simple, since serial data is slow to arrive, and the Arduino can read it faster than it can arrive, as long as the Arduino is doing anything stupid like delay().

Will increased polling really solve the issue as all this data comes in a single blast?

No, it does NOT come “in a single blast”. Serial data comes one character at a time. It CAN be read far faster than it arrives.

Or is their a way that as soon as RX data is being received we tell the micro to focus on getting it all then once all is received continue normal execution?

“Normal execution” should be to read serial data often.

I do make use of delay in my loop, and now that I think about it you make a good point, Serial data is very slow in comparison to the 20Mhz operating speed of the Arduino. :roll_eyes:

Im stuck in the sequential programming paradigm and I believe the solution would be to move to an event driven model without the use of delay.

I will implement this and report back. Thank you for pointing out the obvious, sometimes I think the problem is far more complex and forget the basics.

in comparison to the 20Mhz operating speed of the Arduino.

Of which Arduino?

Im using a ATmega1284P on breadboard using the "mighty 1284p" board.

The 2nd and 3rd examples in Serial Input Basics receive the data as it arrives and place it into a char array. In the examples the array is 32 bytes, but you can make it as big as you need, or memory allows.

There is no need to modify the Arduino serial input buffer.

...R

you either have to increase the receive buffer size to match the largest block of data you expect to receive, or use hardware flow control.

arduino that use avr chips do not support hardware flow control.

I have used teensy 3.2 and enabled the arm processor hardware flow control and able to transfer data between teensy and esp8266 at max speed of 4608000 baud.

doughboy:
you either have to increase the receive buffer size to match the largest block of data you expect to receive, or use hardware flow control.

I don’t agree.

An Arduino can empty the input buffer much faster than data can arrive. That is the basis of the examples in Serial Input Basics

…R

Robin2:
I don't agree.

An Arduino can empty the input buffer much faster than data can arrive. That is the basis of the examples in Serial Input Basics

...R

lol, if all you do in a simpleton program is just read from serial buffer, sure. But no useful real world program does that only. The serial library design is data will be dropped if rx buffer is full. This problem occurs at 115200 baud (hence this thread). I needed to do a really high throughput data transfer at esp8266 max supported baud rate of 4.6megabaud so hardware handshake is mandatory in order to not lose data.

doughboy:
lol, if all you do in a simpleton program is just read from serial buffer, sure.

That's what functions are for. If the code is nicely organised the function that collects the data can keep the buffer from overflowing while other functions deal with other stuff.

I don't see anything "simpleton" about that.

I admit I have no experience at 4.6 MegaBaud - the Uno maxes out at 1 MB and I had no problem with that.

...R

Someone in another forum made the calculations on how much time you have to execute other code and not read the rx buffer before it will overflow, assuming a constant inflow at a given baud rate. This decreases as the baud rate goes higher. The the reality is, you cannot always guarantee with certainty that a function will not block execution > the limit where rx buffer will fill up. The only way to guarantee with absolute certainty to have no data loss is to use hardware handshake. Since AVR does not support it, the next best approach is to keep received data low (like not use the esp8266 to list all wifi access points), and to try to process the rx buffer as soon as possible (but this is still just best effort and does not guarantee no loss of data). To OP, if you do not require speed, you can try using a lower baud rate as that gives the arduino more time to process the rx buffer data. I know esp8266 processing at 115200 baud will result in frequent dropped characters. This was the whole reason I had to create the hardware handshake solution, as espressif engineers informed me that is the only solution that will not lose data.

Someone in another forum made the calculations...

I'm sure this may have happened in other forums, as it is a good way to think about the system througput, but it has also been made here, numerous times (e.g. here).

The only way to guarantee with absolute certainty to have no data loss is to use hardware handshake... as espressif engineers informed me that is the only solution that will not lose data.

You'll forgive me if I disagree, but there is absolute certainty in very few things, including handshaking. Since the Arduino does not implement HardWare handshaking, you can only do the handshaking with the SoftWare, albeit with a HW line. And so it is not quite absolutely certain that the Arduino will toggle that HW line in time. You must still coordinate other things in the Arduino so that it is able to do that handshaking in a timely fashion.

Second, I must disagree that this is the "only way" to avoid data loss. If you could handle the character during the RX interrupt, in a timely fashion, you could avoid the buffer overflow altogether. This is possible when raw data forwarding has a higher throughput on the send side, or if the data can be quickly parsed into a smaller form (e.g., text into numeric values). This is what led me to add an attachInterrupt method to HardwareSerial et al. I no longer have to worry about getting back to loop in time to do a Serial.read.

Still, 4.6Mb is impressive. What was your effective throughput?

Cheers,
/dev

/dev:
I'm sure this may have happened in other forums, as it is a good way to think about the system througput, but it has also been made here, numerous times (e.g. here).
You'll forgive me if I disagree, but there is absolute certainty in very few things, including handshaking. Since the Arduino does not implement HardWare handshaking, you can only do the handshaking with the SoftWare, albeit with a HW line. And so it is not quite absolutely certain that the Arduino will toggle that HW line in time. You must still coordinate other things in the Arduino so that it is able to do that handshaking in a timely fashion.

Second, I must disagree that this is the "only way" to avoid data loss. If you could handle the character during the RX interrupt, in a timely fashion, you could avoid the buffer overflow altogether. This is possible when raw data forwarding has a higher throughput on the send side, or if the data can be quickly parsed into a smaller form (e.g., text into numeric values). This is what led me to add an attachInterrupt method to HardwareSerial et al. I no longer have to worry about getting back to loop in time to do a Serial.read.

Still, 4.6Mb is impressive. What was your effective throughput?

Cheers,
/dev

Your ideas still cannot not guarantee no loss of data.
Even if you handle the rx interrupt, you will still need to process the data outside of ISR, and without hardware handshake to tell sender to stop sending, sender will simply continue to dump data, and sooner or later, can be lost.

You are correct that doing hardware handshake in software does not guarantee no loss of data, as the data in the register can be lost.

at 4.6 megabaud on esp8266, I get at least 30x improvement or better compared to 115200. So something that takes 30 seconds to transfer at 115200 will take 1 second or less at 4.6Mb.

This is what rx looks like using rts hardware handshake

doughboy:
Your ideas still cannot not guarantee no loss of data.
Even if you handle the rx interrupt, you will still need to process the data outside of ISR, and without hardware handshake to tell sender to stop sending, sender will simply continue to dump data, and sooner or later, can be lost.

You are correct that doing hardware handshake in software does not guarantee no loss of data, as the data in the register can be lost.

Just Curious, I implemented RTS/CTS by modifying HardwareSerial and adding GreyGnomes EnableInterrupt library, how did you implement RTS/CTS?

chuck.