The size of the input buffer when reading from an Arduino board

Community General Discussion
1

Dear Community,

Though not Arduino-specific I think it is a crucial thing to understand when working on the communication with a PC, and therefore might be useful for others.

Suppose I have the following setup. An Arduino board running the following code connected to a linux PC.

void setup()
{
    Serial.begin( 38400 );
}

unsigned n = 0;

void loop()
{
    Serial.println("abracadabra 00000000000000000000000000000000000000000");
    Serial.println(n);
    n++;
}

Suppose now I'm reading that output on my PC using the following Python script:

#!/usr/bin/env python3

import serial
import time

s = serial.Serial('/dev/ttyACM0', baudrate=38400, timeout=0.1)

def read_arduino() :
    l = str(s.readline().strip())
    print(l)

    if "abracadabra" in l:
        current = int(s.readline().strip().decode("ascii"))
        print(current)
    else :
        print("Error: no key found")

time.sleep(3) # wait for the "buffer" to fiil

while True :
    print(s.inWaiting()) # ~4000
    time.sleep(0.1)
    read_arduino()

So I have the following case: the receiver (PC) reads the output slower than the sender (Arduino) writes. So data must be lost somehow. To confirm this we can run the python script and see the output. I see the following:

...
64
b'abracadabra 00000000000000000000000000000000000000000'
65
b'abracadabra 00000000000000000000000000000000000000000'
66
b'abracadabra 00000000000000000000000000000000000000000'
67
b'abracadab0adabra 00000000000000000000000000000000000000000'
Error: no key found
b'537'
Error: no key found
b'abracadabraacadabra 00000000000000000000000000000000000000000'
536
b'abracadabra 00000000000000000000000000000000000000000'
537
b'abracadabra 00000000000000000000000000000000000000000'
538
b'abracadabra 00000000000000000000000000000000000000000'
539
...

So I see that there is data loss between n = 67 and n = 537. The number of bytes transmitted from n = 0 to n = 67 is about 4000. The Serial.inWaiting function returns 4010 or something so I think the behaviour is the following: the serial.Serial allocates a buffer of 4kB on the instantiation (port opening ?). But I don't know it for sure. More over, on another PC (also running linux, but different, older, kernel) I have the same result from inWaiting (about 4000) but the same experiment shows many more bytes received --- about 80kB. So I have a question: is my reasoning correct, and if so how do I know the actual buffer size available ?

Thank you in advance !

2

Have you considered simply Googling your question instead of trying to approximate it indirectly? Given the open source nature of most Linux distros, it should be easy to find. This is the first hit I get: c - How to change the size of the system buffer in linux for uart? - Stack Overflow
linux tty uart buffer size - Google Search

It's apparently 4096 bytes.

3

Changing the buffer size will only move the bar :smiley: Make it 8k and the problem starts there, make it 1M and the problem starts there.

If you need to prevent data loss, you will need to implement handshake and/or protocols.

Hardware handshake will be difficult with the Arduino connected via USB. Software handshake with Xon/Xoff can be implemented if the data is ascii.

Protocols can work as below (to give you the idea)

  1. Arduino side
    • Arduino sends a packet of data.
    • Arduino waits for an acknowledge from the PC
    • Back to first step.
  2. PC side
    • PC reads a complete packet based on length or a specified (optional start-marker) and an end-marker.
    • PC acknowledges receipt of the packet.
      If processing takes a long time, acknowledgement should happen after the data has been processed.
4

Thank you all for your replies, but my question is not about how to make it work. I wouldn't do that (data loss is clear here) I'm writing a protocol implementation f.e. here for my personal needs. Sorry if I wasn't clear. Let me explain.

I was asked to look at the code which sometimes doesn't work as expected. The process lead me to the experiment I posted.

Have you considered simply Googling your question instead of trying to approximate it indirectly?

Of course, I have.

It's apparently 4096 bytes.

I thought so, and it seems so on my laptop. But another one (the work one) shows different results:

...
b'abracadabra 0000000000000000000000000000000000000000000'
1316
b'abracadabra 0000000000000000000000000000000000000000000'
1317
b'abracadabra 0000000000000000000000000000000000000000000'
1318
b'abracadabra 000000000000000000000000000000000000000000000000000000'
1340
b'abracadabra 0000000000000000000000000000000000000000000'
1341
b'abracadabra 0000000000000000000000000000000000000000000'
1342
b'abracadabra 0000000000000000000000000000000000000000000'
1343
b'abracadabra 0000000000 0000000000000000000000000000000000000000000'
1365
...

And the data is corrupted/lost after n = 1317. That's why I am asking :slight_smile:

I admit my question might be Python-specific and shouldn't have been asked here. I just thought that people probably often use pySerial and has the knowledge which I don't :slight_smile: . Maybe I will ask on SE.

Thank you all !

5

You could root around in the documentation of Linux distro that runs on that laptop to see if maybe it has a different buffer size.

Either way, I think the net conclusion is always going to be that it's problematic to rely on very large UART buffers, so @sterretje's response above is very much to the point.

Well, you want to make something work, I gather. Otherwise the question is a bit theoretical/academic, isn't it?

6

Will try, yes.

Absolutely agree !

Yes, thank you @sterretje ! :slight_smile:

7

It could be complicated. Hardware might provide some buffering. Uart driver (and/or USB), OS, and Python, might EACH have additional buffering…