Serial Read Data Loss With Arduino Nano

Hi everyone.

I am trying to use Serial reading from a python serial code (which is running on my Jetson Nano) in my project, but in some loop in my code, my data is receiving wrong.

  • I use Serial.readBytesUntil() function for interrupt-like reading.
  • I use standart blue mini-USB cable for communication.
  • My Arduino Nano also connected a nRF24L01 module with SPI.
  • Baudrate is 9600 for both devices. I tried 115200 but there is no difference.
  • The data loss is about %10 of all loops with both test codes (image-1, image-2).
  • There is no positive difference when I use RX/TX pins on both boards with level shifter.
  • When I use the test code in my main code with USB connection, the data loss increases %50 of all loops. But Arduino serial monitor is fine like image-3.
  • When I use the test code in my main code with RX/TX pins, Arduino serial monitor looks like image-4 and Arduino IDE gives error like image-5.

Here is my receiver test code on my Arduino Nano.

 void setup() {

  Serial.begin(9600);
  pinMode(LED_BUILTIN, OUTPUT);
  while (!Serial) {}

}

void loop() {
  char buffer[4];
  int deger = 0;
  if (Serial.available() > 0) {
    int size = Serial.readBytesUntil('\n', buffer, 4);
    
    if (size == 4) {
      Serial.print("deger: ");
      Serial.write(buffer);
      Serial.print("  size: ");
      Serial.print(size);

      // Bytes to integer
      deger = (buffer[1] - 48) * 100 + (int(buffer[2]) - 48) * 10 + (buffer[3] - 48);
      if (buffer[0] == '1') {
        deger = -1*deger; }
      
      Serial.print(" Sayi: ");
      Serial.println(deger);
      }
    
    if (deger < 0) {
      digitalWrite(LED_BUILTIN, HIGH);
    }
    else {
      digitalWrite(LED_BUILTIN, LOW);
    }
  }
}

And the python serial transmitter test code.

 import serial

ser = serial.Serial('/dev/ttyUSB0',baudrate=9600,timeout=10)
while True:
    hata = -58

    if hata > 0:
        hata = str(hata)
        if len(hata) == 2:
            hata = '00' + hata
        elif len(hata) == 1:
            hata = '000' + hata
        else:
                hata = '0' + hata    #Pozitive, Added 0
    elif hata < 0:
        hata = -1 * hata
        hata = str(hata)
        if len(hata) == 2:
            hata = '10' + hata
        elif len(hata) == 1:
            hata = '100' + hata
        else:
            hata = '1' + hata    #Negative, Added 1
    else:
        hata = str(hata)
        hata = '000' + hata

    # For every loop, sends 4 character and "\n"  Example: 1058\n
    gonder = hata + '\n'       # For readBytesUntil function
    ser.write(gonder.encode('utf-8'))
    print(gonder.encode('utf-8'))

image-1: Arduino test code serial monitor screenshot

image-2: Python test code serial terminal screenshot

image-3: Arduino main project code monitor with USB cable screenshot

image-4: Arduino main project code monitor with RX/TX pins on GPIO screenshot

image-5: Arduino main project code IDE error

So, what did i do wrong?
Thank you!

(This is my first new topic. Forgive me if i do wrong about new topic rules.)

image-1-ttyUSB-test-arduino.png

image-2-ttyUSB-test-python.png

image-3-ttyUSB-main-arduino.png

You are sending five bytes constantly from Python. On the Arduino side, you're printing far more than five characters, so your serial output buffer will become full and block.

While it's blocking, you're still receiving and eventually I expect your Serial receive buffer will fill up too. When it does, any more data that arrives from Python will be discarded.

As a test, put a delay in the python loop.

The serial input basics tutorial may be of interest.

Serial.write(buffer);

This is not correct in that you do not have a null terminated character array with this reading code.

char buffer[4];
 int size = Serial.readBytesUntil('\n', buffer, 4)

You have been lucky in that the next memory location after the 4 bytes of buffer is empty.

Have a look at this Simple Python - Arduino demo

The Python code should work on Windows if you edit it to use the Windows style of COM ports.

...R

It would be much faster and reliable to interface Python with your Arduino using the compatible libraries: pySerialTransfer and SerialTransfer.h.

pySerialTransfer is pip-installable and cross-platform compatible. SerialTransfer.h runs on the Arduino platform and can be installed through the Arduino IDE’s Libraries Manager.

Both of these libraries have highly efficient and robust packetizing/parsing algorithms with easy to use APIs.

Example Python Script:

import time
from pySerialTransfer import pySerialTransfer as txfer


if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM17')
        
        link.open()
        time.sleep(2) # allow some time for the Arduino to completely reset
        
        while True:
            send_size = 0
            
            ###################################################################
            # Send a list
            ###################################################################
            list_ = [1, 3]
            list_size = link.tx_obj(list_)
            send_size += list_size
            
            ###################################################################
            # Send a string
            ###################################################################
            str_ = 'hello'
            str_size = link.tx_obj(str_, send_size) - send_size
            send_size += str_size
            
            ###################################################################
            # Send a float
            ###################################################################
            float_ = 5.234
            float_size = link.tx_obj(float_, send_size) - send_size
            send_size += float_size
            
            ###################################################################
            # Transmit all the data to send in a single packet
            ###################################################################
            link.send(send_size)
            
            ###################################################################
            # Wait for a response and report any errors while receiving packets
            ###################################################################
            while not link.available():
                if link.status < 0:
                    if link.status == -1:
                        print('ERROR: CRC_ERROR')
                    elif link.status == -2:
                        print('ERROR: PAYLOAD_ERROR')
                    elif link.status == -3:
                        print('ERROR: STOP_BYTE_ERROR')
            
            ###################################################################
            # Parse response list
            ###################################################################
            rec_list_  = link.rx_obj(obj_type=type(list_),
                                     obj_byte_size=list_size,
                                     list_format='i')
            
            ###################################################################
            # Parse response string
            ###################################################################
            rec_str_   = link.rx_obj(obj_type=type(str_),
                                     obj_byte_size=str_size,
                                     start_pos=list_size)
            
            ###################################################################
            # Parse response float
            ###################################################################
            rec_float_ = link.rx_obj(obj_type=type(float_),
                                     obj_byte_size=float_size,
                                     start_pos=(list_size + str_size))
            
            ###################################################################
            # Display the received data
            ###################################################################
            print('SENT: {} {} {}'.format(list_, str_, float_))
            print('RCVD: {} {} {}'.format(rec_list_, rec_str_, rec_float_))
            print(' ')
    
    except KeyboardInterrupt:
        link.close()
    
    except:
        import traceback
        traceback.print_exc()
        
        link.close()

Example Arduino Sketch:

#include "SerialTransfer.h"


SerialTransfer myTransfer;


void setup()
{
  Serial.begin(115200);
  myTransfer.begin(Serial);
}


void loop()
{
  if(myTransfer.available())
  {
    // send all received data back to Python
    for(uint16_t i=0; i < myTransfer.bytesRead; i++)
      myTransfer.txBuff[i] = myTransfer.rxBuff[i];
    
    myTransfer.sendData(myTransfer.bytesRead);
  }
}

On the Arduino side, you can use myTransfer.txObj() and myTransfer.rxObj() to copy values to the library’s RX buffer and parse multi-byte variables out of the library’s TX buffer.

For theory behind robust serial communication, check out the tutorials Serial Input Basics and Serial Input Advanced.

Thanks to everyone.

wildbill:
You are sending five bytes constantly from Python. On the Arduino side, you're printing far more than five characters, so your serial output buffer will become full and block.

While it's blocking, you're still receiving and eventually I expect your Serial receive buffer will fill up too. When it does, any more data that arrives from Python will be discarded.

As a test, put a delay in the python loop.

I increased the buffer size like you said. Now, its working well with my test codes. It's still working with data loss in my main project code but i hope i will find a solution.

I'm a beginner on digital communication. I will read all documents about serial communication on this topic. It's not so easy without feedback. So, I thank you all.

dcwaves:
It's still working with data loss in my main project code but i hope i will find a solution.

You have a solution in Reply #5.

Power_Broker:
It would be much faster and reliable to interface Python with your Arduino using the compatible libraries: pySerialTransfer and SerialTransfer.h.

pySerialTransfer is pip-installable and cross-platform compatible. SerialTransfer.h runs on the Arduino platform and can be installed through the Arduino IDE’s Libraries Manager.

Both of these libraries have highly efficient and robust packetizing/parsing algorithms with easy to use APIs.

Example Python Script:

import time

from pySerialTransfer import pySerialTransfer as txfer

if name == ‘main’:
   try:
       link = txfer.SerialTransfer(‘COM17’)
       
       link.open()
       time.sleep(2) # allow some time for the Arduino to completely reset
       
       while True:
           send_size = 0
           
           ###################################################################
           # Send a list
           ###################################################################
           list_ = [1, 3]
           list_size = link.tx_obj(list_)
           send_size += list_size
           
           ###################################################################
           # Send a string
           ###################################################################
           str_ = ‘hello’
           str_size = link.tx_obj(str_, send_size) - send_size
           send_size += str_size
           
           ###################################################################
           # Send a float
           ###################################################################
           float_ = 5.234
           float_size = link.tx_obj(float_, send_size) - send_size
           send_size += float_size
           
           ###################################################################
           # Transmit all the data to send in a single packet
           ###################################################################
           link.send(send_size)
           
           ###################################################################
           # Wait for a response and report any errors while receiving packets
           ###################################################################
           while not link.available():
               if link.status < 0:
                   if link.status == -1:
                       print(‘ERROR: CRC_ERROR’)
                   elif link.status == -2:
                       print(‘ERROR: PAYLOAD_ERROR’)
                   elif link.status == -3:
                       print(‘ERROR: STOP_BYTE_ERROR’)
           
           ###################################################################
           # Parse response list
           ###################################################################
           rec_list_  = link.rx_obj(obj_type=type(list_),
                                    obj_byte_size=list_size,
                                    list_format=‘i’)
           
           ###################################################################
           # Parse response string
           ###################################################################
           rec_str_   = link.rx_obj(obj_type=type(str_),
                                    obj_byte_size=str_size,
                                    start_pos=list_size)
           
           ###################################################################
           # Parse response float
           ###################################################################
           rec_float_ = link.rx_obj(obj_type=type(float_),
                                    obj_byte_size=float_size,
                                    start_pos=(list_size + str_size))
           
           ###################################################################
           # Display the received data
           ###################################################################
           print(‘SENT: {} {} {}’.format(list_, str_, float_))
           print(‘RCVD: {} {} {}’.format(rec_list_, rec_str_, rec_float_))
           print(’ ')
   
   except KeyboardInterrupt:
       link.close()
   
   except:
       import traceback
       traceback.print_exc()
       
       link.close()





**Example Arduino Sketch:**


#include “SerialTransfer.h”

SerialTransfer myTransfer;

void setup()
{
 Serial.begin(115200);
 myTransfer.begin(Serial);
}

void loop()
{
 if(myTransfer.available())
 {
   // send all received data back to Python
   for(uint16_t i=0; i < myTransfer.bytesRead; i++)
     myTransfer.txBuff[i] = myTransfer.rxBuff[i];
   
   myTransfer.sendData(myTransfer.bytesRead);
 }
}





On the Arduino side, you can use `myTransfer.txObj()` and `myTransfer.rxObj()` to copy values to the library's RX buffer and parse multi-byte variables out of the library's TX buffer.


For theory behind robust serial communication, check out the tutorials [Serial Input Basics](https://forum.arduino.cc/index.php?topic=396450.0) and [Serial Input Advanced](https://forum.arduino.cc/index.php?topic=662346.0).

Thank you. I’m trying to use this library for high performance like you said. I can’t understand the structure yet.

How do i use for data receiving for Arduino side? Is it about byte counting? Can you give me an example for one string and one float receiving and copying in one loop?