Serial communication with Python - beginner qestion

I'm taking my first steps in using serial port to pass data from my arduino board to Python. I'm running into difficulties and would appreciate if someone can tell me where I'm going wrong. Explanation of my issue is at the end of this post.

The board is Sparkfun Redboard
I'm using Python 3 on a Windows 10 PC.

Arduino code:

int aiPin = A0;
int data;

void setup() {
  Serial.begin(9600);
  pinMode(aiPin, INPUT);
}

void loop() {
  data = analogRead(aiPin);
  Serial.println(data);
  delay(50); 
}

Python code:

import serial
import time

ArduinoSerial = serial.Serial('COM3',9600) 
time.sleep(2) 

b = ArduinoSerial.readline()
print(b)
s1=  b.decode() # convert to string
print(s1)
s2=  s1.rstrip()  # strip away excess characters
print(s2)

ArduinoSerial.close() 

This works fine if the delay in my Arduino loop is above 10ms. In Python, b will come in is as something like b'381\r\n' and the decode method can be used to convert this to a string. s2 will be '381'.

If I reduce the delay in my Arduino code to 10ms or less (or remove the delay entirely), I start to run into problems. In this case b will come in as something like b'\x00\x00\x00\x00\xff433\r\n' and the decode method in Python won't work. It will lead to the following error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 4: invalid start byte

I don't understand:

a) Why delay in the Arduino loop is affecting what I read in Python
b) If there is something I can do in the Arduino code to prevent this (other than have a slow loop rate)
c) If there is something I can do in the Python code

Any help much appreciated!

No need to be in a hurry to close the serial port leave it open while you read in all of the data.

Using a while loop in Python will constantly monitor for incoming data, no need to use sleep. Try this example.

import serial

ser = serial.Serial("COM7",9600,timeout=1)


while (True):

	if ser.isOpen():

		input_data=ser.readline().strip().decode("utf-8")
		print(input_data)

There is more work to do with this such as running this code in its own thread and a little bit of error checking but it will give you the confidence that you are able to receive the Arduino data

Thanks.

However, I have the same experience with this code as I did with mine.

If I slow down the Arduino loop it runs fine.

However if I put the Arduino loop delay at 10ms or less (or remove the delay) it starts to fail on the Python side. I get the same sort of error as I got previously, e.g.

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte

When it fails, it always fails on the first iteration of the Python loop. Occasionally it works on the first iteration and when this happens it continues looping without error.

I had a previous version of my Python code with loops and the the consistent failing on the first iteration was the reason I put a sleep in my original code after creating the serial port object. I thought there might need to be a bit of a delay before the first read.


Potential solution:

I have been playing around a bit more and have discovered that immediately closing and then opening the port after creating the serial object seems to lead to error free execution even if Arduino loop is running as fast as possible (no delay command in Arduino loop).

So, in Python:

ser = serial.Serial("COM3",9600)
ser.close()
ser.open()

..rest of code

Any idea why this appears to help?

I guess it's something to do with what's not getting cleared from the serial buffers when you disconnect/reconnect I'm really not sure.

I have something here that's not really a remedy but it's the exception handler for the error you were getting, at least while you work on building your project it may allow you to continue to monitor your data beyond the initial errors you were getting, it does no harm to have it in there. Also enabling DTR at start up can help by resetting the Arduino, if it works on your board.

ser = serial.Serial()

ser.dtr=True
ser.baudrate=(9600)
ser.port="COM3"
ser.open()

while (True):

	if ser.isOpen():

		try:

			input_data=ser.readline().strip().decode("utf-8")
			print(input_data)

		except UnicodeDecodeError as e:

			print(e)

Well this problem has provided a few days of entertainment/frustration. When I first looked at it I thought "That can't be right - I have never seen this problem". I downloaded the code and the first couple of time I didn't see the problem. A few more tries and it intermittently showed up - more often then not. I tried a variety of things (some still commented out in the code) and finally figured out it is a boot loader problem.
Whenever the Arduino is reset the boot loader is called which tries a handshake over the serial port to see if there is new code to upload. If not it then goes to the last loaded program. Depending on the timing, the Python code may read the end of the handshake which apparently is not valid ASCII (particularly if the baud rates are different).
The default Python serial does not use hardware flow control, but apparently the USB to serial set up does. Watch the lights when the Python code is started and you will see the lights blink as the boot loader is called.
I have appended the final Arduino and Python code for you amusement. I commented out the analogue read as it turned out to be irrelevant. To end the Python program press Ctrl-c.
If you press the reset button on the Arduino while the Python code is running you will see an error on every reset.

int aiPin = A0;
int data = 600;

void setup() {
  Serial.begin(9600);
 // delay(100); 
  pinMode(aiPin, INPUT);
}

void loop() {
//  data = analogRead(aiPin);
  Serial.println(data);
  //delay(50); 
}

Python code

import time
import serial
import serial.tools.list_ports

#
# Find the USB port we are on
#
commports = serial.tools.list_ports.comports() # get possible ports
numPorts = len(commports)
if (numPorts == 0):
    print("No serial ports available\n\n")
    exit()
if (numPorts>1):
    # Have user pick one
    portNum = 0
    for port in commports:
        print("port number ",portNum)
        print(port)
        portNum = portNum+1
    usePort = int(input('enter port number to use 0-'+str(numPorts-1)+':'))
else:
    usePort = 0

thePort = commports[usePort][0]
print('using ',thePort,'\n')

# open serial port
#device = serial.Serial(thePort, 9600)
device = serial.Serial()
device.baudrate= 9600
device.port=thePort
device.open()
#time.sleep(5)
#device.reset_input_buffer()
#input_data=device.readline()
i=0
while(True):
    if device.isOpen():
        i +=1
#         input_data=device.readline().strip().decode("utf-8")
        try:
          input_data=device.readline()
 #       print(input_data)
          s1 = input_data.strip().decode("utf-8")
          if (s1 != "600"):
             print("Miscompare" , input_data)    
  #      print(s1)
   #    print(s2)
        except UnicodeDecodeError as e:
           print(e)
           print(input_data)
        except KeyboardInterrupt:
          print("\nexiting program ",i," tests")
          device.close()
          exit(0)

Good job oldcurmudgeon, it also had me pondering, I knew there were unwanted bytes and it appeared to me the number would vary each time. Although I didn't know where these unwanted bytes were coming from, until you explained it, I did come up with a slightly different solution which I will show for comparison. The idea is to discard all the unwanted bytes that might give "decode" a problem by using "read_until" and using a short string to indicate that "clean data" is about to commence.

So after initializing the serial I transmit a small string, when python receives this string all the unwanted bytes will have been read and discarded.

void setup()
{
  
  Serial.begin(9600);  
 
  Serial.print("<>"); 

  pinMode(aiPin, INPUT);
  
}

At the python end I use read_until

ser = serial.Serial(timeout=5)
ser.rts=True
ser.baudrate=(9600)
ser.port="COM3"

ser.open()

start_signal = "<>".encode()

ser.read_until(start_signal)

Of course this depends on the Arduino being reset before each exchange of data

It's basically what Robin does in his Serial Input Basics - updated tutorial; not necessarily for the same reason. His text is a bit longer though.

Does the problem occur when the Arduino is reset (with the button) or when it is reset because you open the serial port?

If the latter and you're using Windows, you can fiddle with the DTR and RTS signals when you open the port to prevent the undesired reset. I'm not a Python programmer so can't quite advise further.

Thanks for continued input and discussion. I probably won't get a chance to properly read, digest, and try the solutions until the weekend, but will do so then.

The boot loader gets called on any reset - power on, pressing the reset button, serial port open, brownout, etc. What is particularly annoying is that it clears the interrupt register so that by the time your code gets control you don't know what caused it. At times it would be useful to know if it was reset due to brownout. Yes, I know the boot loader can be modified to "fix" this. The "fix' is in quotes because they do have reasons for what they do.

For that I gave you a possible solution.

Get rid of the bootloader :smiley: You can program the RedBoard over ICSP (although on my one the pins are missing).

I'm not sure if your problem is RedBoard specific. I've seen the behaviour on the RedBoard, but as far as I remember not on original Unos.

I was using an Uno R3, I just dug out my original Duemilanove and it has the same problem. I suspect it is common across the 8 bit Arduinos.

Yes, bagging the boot loader and loading direct is a (possible) solution. I have a couple of ICSP programmers and have used that option on occasion. If you are working with beginners is it not a great solution. It isn't very hard but it is tedious and one more thing to worry about when the newcomer is trying to get something to work.

With 8U2 / 16U2 or with another TTL-to-USB converter.

That uses the FT232RL which is the same as on the SparkFun RedBoard.

Just trying to figure out possible culprits.

What I am seeing is not a handshake from the boot loader but serial data transmitted from within the Arduino loop. Because the Arduino sketch is still running and still trying to transmit there is a brief time between opening a serial port and the software reset where a small packet of data is transmitted. Having no delay in the loop makes this a lot more noticable. I tried this by using an elapsed time value in the Serial.println, if I closed the serial port at 30 seconds and then reopened 30 seconds later the first 7 to 10 or more values I received were in excess of 1 minute then the values would start incrementing from zero as expected. Also within these values, because of the abrupt closing and opening of the serial port, were a few bytes that seem to have been corrupted which is what caused the errors for the OP. In some situations it would be unnoticeable but being aware of what is happening makes it a simple issue to deal with. All of the above was done with a genuine Arduino Uno communicating over USB so other methods may be different.

My R3 is actually a clone which uses a CH340 variant. I got out a Mega which uses a 16U2, it behaves the same way.

To test this I used a separate USB to serial cable with only the tx and ground connected to the Arduino. This would eliminate the possibility of a serial open reset. For power I used a wall wart connected to the barrel connector. Same result when running the Python program multiple times or pressing the Arduino reset button. You are right - the problem occurs even without the boot loader. I may owe the boot loader an apology :slightly_smiling_face:.

As a heads up, you might be interested in streamlining the data transfer process between your Arduino and Python programs by using the compatible libraries SerialTransfer.h and pySerialTransfer. SerailTransfer.h can be installed via the Arduino IDE's Libraries Manager and pySerialTransfer can be pip installed. Both come with many examples like the following:

Arduino TX Data:

#include "SerialTransfer.h"


SerialTransfer myTransfer;

struct STRUCT {
  char z;
  double y;
} testStruct;

char arr[] = "hello";


void setup()
{
  Serial.begin(115200);
  myTransfer.begin(Serial);

  testStruct.z = '$';
  testStruct.y = 4.5;
}


void loop()
{
  // use this variable to keep track of how many
  // bytes we're stuffing in the transmit buffer
  uint16_t sendSize = 0;

  ///////////////////////////////////////// Stuff buffer with struct
  sendSize = myTransfer.txObj(testStruct, sendSize);

  ///////////////////////////////////////// Stuff buffer with array
  sendSize = myTransfer.txObj(arr, sendSize);

  ///////////////////////////////////////// Send buffer
  myTransfer.sendData(sendSize);
  delay(500);
}

Python RX Data:

from time import sleep
from pySerialTransfer import pySerialTransfer as txfer


class struct(object):
    z = ''
    y = 0.0


arr = ''


if __name__ == '__main__':
    try:
        testStruct = struct
        link = txfer.SerialTransfer('COM11')
        
        link.open()
        sleep(5)
    
        while True:
            if link.available():
                recSize = 0
                
                testStruct.z = link.rx_obj(obj_type='c', start_pos=recSize)
                recSize += txfer.STRUCT_FORMAT_LENGTHS['c']
                
                testStruct.y = link.rx_obj(obj_type='f', start_pos=recSize)
                recSize += txfer.STRUCT_FORMAT_LENGTHS['f']
                
                arr = link.rx_obj(obj_type=str,
                                  start_pos=recSize,
                                  obj_byte_size=6)
                recSize += len(arr)
                
                print('{}{} | {}'.format(testStruct.z, testStruct.y, arr))
                
            elif link.status < 0:
                if link.status == txfer.CRC_ERROR:
                    print('ERROR: CRC_ERROR')
                elif link.status == txfer.PAYLOAD_ERROR:
                    print('ERROR: PAYLOAD_ERROR')
                elif link.status == txfer.STOP_BYTE_ERROR:
                    print('ERROR: STOP_BYTE_ERROR')
                else:
                    print('ERROR: {}'.format(link.status))
                
        
    except KeyboardInterrupt:
        link.close()