Arduino, Pyserial - Only reading data on first Python Execution

[adamgronewold]

Hello and thanks in advance for the help. My problem is similar to others described on the forum, but these discussions are old, have since become inactive, and a reliable solution to my problem doesn't seem to exist.

I am trying to perform basic serial communication between an Arduino and a Linux machine via USB on the DUE programming port. After connecting the Arduino to my Linux machine, I check to make sure the device exists under 'lsusb' and 'ls /dev/ttyACM*'. Once connected, I run a python script to open the appropriate serial port '/dev/ttyACM1' and I attempt to read data arriving from the Arduino and print it out, line by line, to the terminal. This works perfectly after I first connect my Arduino to the Linux machine.

I then terminate my python script and run it again. Suddenly the data is no longer visible on the Linux side. The device remains connected under 'lsusb' and 'ls /dev/tty*', but attempting to print the arriving data only gives b' '. Ideally, I'd like to resolve this issue for my Arduino DUE, however I have tried other DUEs, and an UNO without resolve.

My minimal code to produce this problem is as follows:

Arduino Side

const int period=8
unsigned long lastTime=0;
void setup() {
    Serial.begin(9600);
    delay(100);
}
void loop() {
    if ((millis() -lastTime)>period) {
        lastTime=millis();
        Serial.println("The message");
    }
}

Basically this code is meant to simple puke data over serial at a set rate of 125 Hz, regardless of whether or not another device is there to read it.

Python/Linux Side

import serial
import time

def main(args=None):

    ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600, timeout=0.01)
    print(ser.isOpen())
    
    while (True):
        try:
            print(ser.readline().decode('utf-8'))
        except UnicodeDecodeError:
            print(ser.readline())
        except KeyboardInterrupt:
            ser.close()
            time.sleep(2)
            print(ser.isOpen())
            time.sleep(2)        
            
if __name__ == "__main__":
    main()

Nothing too crazy here. As mentioned, the python side reads the Arduino's data perfectly after I first connect the Arduino. In the case above python will just print "The message" to terminal repeatedly. But, if I close with 'Ctrl+C' and rerun my python from terminal all I will see is blank lines (or b' ' if I remove the decode part of my readline). I have tried a variety of methods to fix the problem, including placing flush() or reset_input_buffer() in various places. ser.isOpen() always returns 1 when the port is open and 0 after I close the port, regardless of whether its the first time executing the code or the second. in_waiting() returns 0 the second execution as well. I am confident this isn't an issue with baudrate, parity, stopbits, etc., because as mentioned, it works on my first execution of the python code.

In fact, I have a Teensy 4.1 I am using basically in the exact same way and it works fine. Only substantial difference is that it is operating on '/dev/ttyACM0'. I can run and interrupt the receiving side python code for this Teensy as many times as I want without an issue. Another forum suggested there is a bug with pyserial that causes my issue, but seeing that I can get this to work on the Teensy tells me there is something else at play here.

I can also see from watching the lights on my DUE that the device is not power cycling when I interrupt the python side code, so in theory it should still be continuing to dump lines of data over serial despite the fact that the port isn't open with Python. I'd like to get to a point where I can receive the serial data without having to unplug the Arduino every time.

Thanks again for your help.

[mikb55]

Replace your Python program with a terminal program such as Minicom.
If the problem disappears then there is a Python issue.

[ptillisch]

Hi @adamgronewold. I gave it a try on my Linux machine, but unfortunately was not able to reproduce the problem you experienced. I receive the expected data even after terminating the Python script and starting it again.

So I'm not able to provide any sort of definitive assistance, but I remember that some previous similar issues ended up being found to be resolved by the correct management of the DTR signal.

For example:

@paulrd

it enables DTR because I've found serial comms with a Leonardo don't work without it.

From a quick glance at the pySerial package's documentation and complete guesswork, I would start by changing this line of your Python script:

@adamgronewold

    ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600, timeout=0.01)

To this:

    ser = serial.Serial(port='/dev/ttyACM1', baudrate=9600, dsrdtr=True, timeout=0.01)

Reference:

https://pyserial.readthedocs.io/en/latest/pyserial_api.html#serial.Serial.__init__

• dsrdtr (bool) – Enable hardware (DSR/DTR) flow control.

[sumguy]

@adamgronewold I agree with @ptillisch on the dtr I think a reset and some way of synchronizing the startup is usually a good idea.

There is a post in the "Introductory Tutorials" that's an interesting read and I would recommend you check it out.

Two ways communication between Python3 and Arduino

Hello everyone, As the use of Python is becoming more widespread in schools, many of you are looking to connect your Arduino to your Mac or PC using Python. Using the serial port of your computer with Python is not complicated, but doing it in a robust way is a bit more challenging if you want to handle the inherent asynchronous operation of serial connections. The purpose of this tutorial is to provide you with minimal code on both the Python and Arduino sides to manage communication in the …

I modified both your Arduino code and Python code to include synchronization.

In the Arduino code there is an extra line just in case the dtr is not having the desired effect.

In Python I modified what happens when the port is opened and I modified how the exceptions are handled. When opening the port you should always see the message "Start"

This is not complete and only intended as something for you to play around with and improve on.

ARDUINO

const int period=8;
unsigned long lastTime=0;
void setup() {
    delay(2000);
    Serial.begin(9600);
    delay(100);
    Serial.println("Start");

}
void loop() {
    if ((millis() -lastTime)>period) {
        lastTime=millis();
        Serial.println("The message");
    }

    if (Serial.available() > 0){Serial.end();setup();}
}

PYTHON

import serial
import time

def main(args=None):

    ser = serial.Serial()
    ser.port='COM5'
    ser.baudrate=9600
    ser.timeout = 0.5
    ser.dtr=True
    ser.open()
    ser.write('\n'.encode())
    
    sync_start = False
    
    while (True):
        if ser.inWaiting():
            try:
                inp=ser.readline().decode('utf-8').strip()
                if 'Start' in inp:
                    sync_start = True
                    print(inp,'\n')
                elif sync_start:
                     print(inp)
                    
            except UnicodeDecodeError as e:
                pass
            except KeyboardInterrupt:
                ser.close()
                break     
            
if __name__ == "__main__":
    main()

EDIT I left the PC com port in there from when I tested the code, obviously you will need to change that

[adamgronewold]

Thank you very much for your response. This did the trick

[ptillisch]

You are welcome. I'm glad it is working now.

Regards,
Per

[adamgronewold]

Sorry for the trouble, but I am actually coming back to this topic after previously marking a solution I thought was correct. It seems it has only made the problem less consistent. Now instead of needing to reconnect the device after the first run, I can run my script many many times before I stop receiving data, and other times I can only run my script once. I've tried tweaking other settings in concert with enabling dsrdtr with no luck.

This is the case with my Arduino Due, the UNO, and with an RTK Facet GNSS/GPS unit. In every case, the serial device is just relaying a constant stream of single line messages with terminating characters. The only device I can cleanly and robustly establish a serial connection is a Teensy 4.1, with or without the dsrdtr option enabled. I should add that my machine is ARM64, although I'm not sure that matters. The behavior is incredibly baffling and frustrating. Any other suggestions from the community?

EDIT
Some more trouble shooting. Using "cat /dev/ttyACM2" or "cat /dev/ttyACM3" (sometimes the device file changes which is fine), I experience the same behavior as the original problem. Running the command the first time will output a constant stream of the messages arriving by serial. When I interrupt the command and rerun the command I am back to no data until I disconnect and reconnect the device (without power cycling the device).

[adamgronewold]

Coming back once more...

I now have the connection also working with the RTK Facet. I know the issue is not:

• dialout permissions
• dsrdtr flow control of lack there of
• pyserial
• python in general
• data left in the serial input buffer, as I have flushed the input buffer in various places before closing, right after opening, etc.

I have reproduced the issue using the serial monitor in the Arduino IDE on my Linux machine. First connection to monitor = good, closing and reopening the monitor = no good. I suspect that the issue is somehow related to how Arduino handles the USB serial connection to my Linux device or vice versa. The device in question is an NVIDIA Jetson Orin Nano ARM64 (Ubuntu 20.04). Teensy devices and the RTK Facet seem to be stable on their respective ports (e.g. /dev/ttyACM*) and unchanging if I powercycle the entire Linux device, but Arduino devices seems to not be stable, often switching the port used when disconnecting and reconnecting the device, even if another device previously existed at another port name. It's as if after closing the serial connection on the Linux side the Arduino thinks its on a different port even if the associated port on the Linux side isn't actually different, or as if the serial connection isn't being properly closed. Isolating the issue such that the only USB serial device is the Arduino doesn't solve the problem.

I'm fresh out of Teensy boards, and I'd really like to stick with Arduino so I can use available headers for the project I am working on.

Thanks again in advance.