Data loss in UART - 115200 Baud ( missing bits)

Hello Everyone,

Thank you in advance for your help! I noticed that when I send packages of different sizes via UART, in the 115200 Baud, I have some missing bits at the end. In the following image this loss is showed:

I've done 10 tests and something weird happened, I lost more bits for the package of 20kb than for the package of 30kb. Can anyone explain me why this happened? Is data loss aleatory, or is there any pattern for this?

I don't know if it helps, but I also plotted a graph showing the percentage of bits that arrived and I got that the 10kb package was the more reliable.

Thank you very much, :grinning:

Danilo

Welcome to the forum.

Without handshake, the receiver has to be fast enough to read every byte.
Can you tell who sends the packages and who receives it.
It there an Arduino board involved ? Which Arduino board ?
Can you show the sketch ?

When I send packages of 20kbyte from a Arduino board to the computer at 115200 baud, then absolutely nothing ever gets lost.

1 Like

post the code you used to prove this... the issue is possibly between the chair and the monitor and looking at the code :slight_smile:

can you share the code?

1 Like

Are You sure about the difference between bits and bytes?

Hi @Koepel! Thank you for the fast response!

I'm doing a research that involves the connection between an Arduino and a Raspberry, for that reason I'm measuring the time that a package takes to go from the Raspberry to the Arduino via UART, I²C and SPI. My go is to compare this guys and evaluate several parameters like: which of them is faster, more reliable, easier to implement and etc.

So for UART, I tried several bauds (2400, 9600, 19200 and 115200) and I only notice loss for the baud of 115200, for the other bauds 100% of the package arrived. So in my head, the greater the package and the baud, the higher would be the data loss. But this didn't happened, the package of 30kb lost fewer bits that the package of 20kb. Is there any sense in what i'm saying?

we understand what you say but we think there is likely a bug in your code and the way you send or receive the data. at 115200 bauds if your wiring is correct and the environment clean, you should have 0 loss on a few k.

Can you please identify the exact UART you are using? I want to read a data sheet for it. Most UARTs have a status register that may give more clues.
Paul

My feeling is that you’re probably overflowing the tx or rx buffer at some point….
possibly not checking the rx buffer often enough - that some data is lost.

Can you check the buffer-overrun flags ?

I really hope you’re not using SoftSerial anywhere in this project.

Cant say about your problem, but thanks for using

aleatory

in your question. Hadda look it up, hope I can slip it into casual conversation from time to time. :neutral_face:

a7

Me too. I've actually learned something today from this forum. :wink:

That’s the beauty of using anglicized foreign words, sometimes they work

Hello!

First I would like to thank you all for the replies and to apologize for the poor explanation

post the code you used to prove this... the issue is possibly between the chair and the monitor and looking at the code :slight_smile: can you share the code?

You are more than right @J-M-L :joy: haha The code that i used in arduino is quite simple and it's shown below


void setup() {

  Serial.begin(115200);
  
}

void loop() {
  
  delay(1000);
  send_package();
}


void send_package() {

  
  Serial.print("A");  //This indicates to Raspberry the start of the data package


 for (int i = 0; i < 1024; i++)  // I want arduino to send 1kb or 1024 bits
    {
      Serial.print("2");  // So I say: "please Arduino send for me the number "2" 1024 times"
    }
       
  Serial.print("Z"); //This indicates to Raspberry the end of the data package

}

So Raspberry on the other side, store all the bytes that arrive in a .txt file (that's how I know if I lost a byte or not) and more than that, when she receive the letter "A" and the letter "Z", I ask her to record the time when they came (thats how I know how long does a full package takes to get from Arduino to Raspberry)

Are You sure about the difference between bits and bytes?

@ Railroader now that you asked me this, when I send this number "2" via UART I'm sending a byte not a bit, isn't that true? I think i mixed it up

Yes I think I'm familiar with the theoretical difference between a byte and a bit

if your wiring is correct and the environment clean

Yes, I do believe that my wiring is correct:

Rasp        Arduino

GPIO 02(SDA) ---- SDA
GPIO 03(SCL) ---- SCL
GND -------- ---- GND

In this site you guys can see the Raspberry pinout and In this site you can see one example of UART between an Arduino and a Raspberry

Can you please identify the exact UART you are using? I want to read a data sheet for it.

Hi @ Paul_KD7HB, I believe that the best place to find information about that is in the datasheet of "Raspberry's Brain" (BCM2837), that uses an ARM architecture.

My feeling is that you’re probably overflowing the tx or rx buffer at some point….
possibly not checking the rx buffer often enough - that some data is lost.
Can you check the buffer-overrun flags ?
I really hope you’re not using SoftSerial anywhere in this project

@lastchancename this may be happening! You have a good point, but as I'm comparing UART, I²C and SPI; wouldn't this overflow make UART less attractive (at least for higher Bauds)?

I bag your pardon, but I'm not familiar with the term "SoftSerial", could you explain it better please? In the raspberry I'm using the serial library. And do you have an Idea on how can I check the buffer-overrun flags ?

@ cattledog and @ alto777 it's nice having you guys here hahah

This is my first topic in the Forum, so If I'm doing something wrong be free to question me

I think here there are some documentation about the Raspberry and probably some good stuff about wich UART she uses. I believe that this information should be in BCM2837 data sheet. Thank you for the help :grinning: :+1:

can you share the RPi code too?

you arduino code could be made even more efficient:

inline void send_package() {
  Serial.write('A');  //This indicates to Raspberry the start of the data package
  for (int i = 0; i < 1024; i++)  // I want arduino to send 1kb or 1024 bits
    Serial.write('2');  // So I say: "please Arduino send for me the number "2" 1024 times"
  Serial.write('Z');  //This indicates to Raspberry the end of the data package
}

void setup() {
  Serial.begin(115200);
}

void loop() {
  delay(1000);
  send_package();
}

Hello,

Maybe you can try to make sure that the buffer in the Arduino does not get overrun. This buffer is situated between your application code and the interrupt handler sending the bytes to your Pi.

void setup() 
{
  Serial.begin(115200);
}

void loop() 
{
  delay(1000);
  send_package();
}


void send_package() {

  
  Serial.print("A");  //This indicates to Raspberry the start of the data package


  for (int i = 0; i < 1024; i++)  // I want arduino to send 1kb or 1024 bits
    {
      Serial.print("2");  // So I say: "please Arduino send for me the number "2" 1024 times"
      while (Serial.availableForWrite()<10)
        delay(10);
    }
       
  Serial.print("Z"); //This indicates to Raspberry the end of the data package

}

that will never happen when writing. Serial.write() or Serial.print() will pause the code until there is room in the outgoing buffer if needed

the question is more likely on the receiving end and how the data gets written to the SD. if that's slow, then the receiving buffer might overflow and then you miss bytes.

given the goal of the test and the very easy to check nature of the code, sending A22222...222Z, there is no point on the RPi side to make it more complex than needed. wait for 'A', count the number of '2' until you get 'Z'. that's all that should be done.

Oh sure! I wrote my code in Python and I hope the comment lines will help you to understand it

# This libraries are used to record time

from datetime import datetime
from time import sleep

# This one is responsible for serial communication
import serial

# Here I define the port (USB0 - for USB cable and ttyS0 for pins GPIO 02 - SDA , GPIO 03- SCL)
# I also define the Baud 115200
ser = serial.Serial('/dev/ttyUSB0',115200) 

ser.flush() #cleans the buffer

# Open the .txt file in write mode
file = open('Outputs/UART_1kb.txt','w')

# This 'for' is here because I want 10 samples
for i in range(10):
    start = False # This guy will be true after I receive "A"
    while True:
        read_var= ser.read() #Reads Serial
               
        if read_var == b'A':
            
            file.write("\n")
          # A strange way to say: "give me the time now with milliseconds"
            file.write(datetime.utcnow().strftime('%H:%M:%S.%f')[:-3])
            file.write("\n")
            start = True
            
        if start == True:
            # "please write the twos that ur receiving "
            file.write(str(read_var.decode("utf-8")))
          
# "Oh if 'Z' came, the package is finished "
        if read_var== b'Z' and start == True:
            file.write("\n")
            file.write(datetime.utcnow().strftime('%H:%M:%S.%f')[:-3])
            file.write("\n")
            break

 # close .txt file
file.close()

@J-M-L thank you a bunch for your suggestion, I'll try to use this code and see if i reduce this package loss

Hey @JOHI thak you for your contribution :slight_smile:

The mini UART has smaller FIFOs. Combined with the lack of flow control, this makes it more prone to losing characters at higher baudrates. It is also generally less capable than a PL011, mainly due to its baud rate link to the VPU clock speed.

The particular deficiencies of the mini UART compared to a PL011 are :

No break detection
No framing errors detection
No parity bit
No receive timeout interrupt
No DCD, DSR, DTR or RI signals

I was able to track down documentation on Raspberry PI UARTs and found this warning/documentation. It is likely the source of your problem.

Paul

I think the issue is more on the reception side. Can you really dumb down the RPi code to wait for 'A', note the time, count the number of '2' until you get 'Z' and note the time again to calculate duration.

1 Like

Re Serial.print()s delaying the loop see my tutorial on Arduino Serial I/O for the Real World
goes into loop delays and adding extra TX/RX buffers and non-blocking Serial output.
My Arduino to Arduino via Serial tutorial my also give you some idea, but not really high speed, more stop and start.

1 Like