Arduino Uno --> Raspberry Pi W/Python 3.4 *** Serial communications w/UART

Hello all,

Once I again I am looking to the collective knowledge of this forum for a solution to my problem.

I am communicating between a Raspberry PI (Python 3.4) and an Arduino Uno, or I should say, trying.

Here is what should happen…

I send a command string to the Arduino “F25000,3500” with a new line character. The Uno reads this byte-by-byte and echos it back to the Pi. The Arduino then processes this data further and waits for a new command.

This works perfectly the first time (First run on the PI’s program, and from a boot on the Arduino. If I just re-run the program over and over again, everything works fine.

However, when the PI sends Two or more commands to the UNO things get strange. Instead of sending the command - it sends odd characters, possibly more than it should. As with the earlier, the first command is fine, just the second, third and so on commands turn to jiberish.

With the first command always working, it seems as though something goes on between commands that messes with the serial data.

THINGS I HAVE NOTICED

  • On first communication, I send 12 bytes and get 12 bytes back (not counting LF)
  • Checking the buffer amount is always 12 on first command
  • Data being Echoed from UNO is identical to data sent
  • On second communication, I send 12 byte and get 18 bytes back
  • Checking the buffer amount is always 18 on the second command
  • Data being echoed from UNO is jiberish

I HAVE TRIED MANY DIFFERENT THINGS TO NO AVAIL, HOWEVER…

  • Assuming it had to do with the serial port and perhaps buffer data on the PI side…
  • I tried flushing the buffers - didn’t help.
  • I tried opening port - sending data - closing port then repeating - didn’t help.
  • Running without the UNO serial port monitor active - didn’t help.
  • Running without the UNO connected to a USB on a computer (power adapter instead - no help.

I suspect the problem is on the PI side but just have not been able to pinpoint the issue.

I sure hope someone can help me out on this one.

Thanks in advance,
Chris D

Python 3.4 code on Raspberry Pi 3

# This routine will send a command to the Arduino and it will execute
# It cannot do it repeatedly within a loop

from time import sleep
import serial
import binascii

err_cnt = 0

#Start up code to open the serial port
ser = serial.Serial("/dev/ttyS0",9600, timeout =1)
print("Hello, waiting for data...")
if ser.isOpen():
    print(ser.name, "is open")
    print(ser)




# Create & send the command to the Arduino
def Send_Command_To_Pi(Command_String, Steps, SPS):
    global err_cnt
    Command_String = Command_String
    ser.write(Command_String.encode('ascii'))
    sleep(.2)
    Val1 = Steps + ","
    ser.write(Val1.encode('ascii'))
    sleep(.2)
    Val2 = SPS
    Check_Val = Command_String + Val1 + Val2
    Val2 = Val2 + "\n"
    ser.write(Val2.encode('ascii'))
    print("Sent")

    sleep(.1)
    Buffer = ser.inWaiting()
    print("Buffer ",Buffer)

    try:
        indata = bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())

        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())  
        print("S",Check_Val)
        print("R",indata)
        print("Cmds ",x, "   Errors ",err_cnt)
    except:
        err_cnt += 1
        print("ERROR count =",err_cnt)
        sleep(1)



#First command sent to Arduino
print("------------------------")
Command_String = "F"
Steps = "25000"
SPS = "3500"
Send_Command_To_Pi(Command_String, Steps, SPS)

#Wait long enough for all activity to be done on Arduino
sleep(20)

#Second command sent to Arduino
print("------------------------")
Command_String = "R"
Steps = "25000"
SPS = "3500"
Send_Command_To_Pi(Command_String, Steps, SPS)

ARDUINO UNO Code (only relevant portions)

...

  Serial.begin(9600);           // set up Serial library at 9600 bps
...

  // Fetch commands from serial port if there are any
  if (Serial.available() > 0) {
    inByte = Serial.read();
    if (inByte != 10 || inByte != 13) 
    {
      Serial.write(inByte); // echo back to pi everything that was sent in this command
    }
    
    // only input if a letter, number, =,?,+ are typed!
    if ((inByte >= 65 && inByte <= 90) || (inByte >= 97 && inByte <= 122) || (inByte >= 48 &&     inByte <= 57) || inByte == 44 || inByte == 61 || inByte == 63) {
      Serial_Command.concat(inByte);
    }// end serial.available
    // Process command when NL/CR are entered:

    // Handle the incomming commands from the Raspberry Pi
    if (inByte == 10 || inByte == 13) {
      inByte = 0;
      // Respond to a command:
      if (Serial_Command.startsWith("F")) //FORWARD MOTION command
      {
        temp1 = Serial_Command.substring(1, Serial_Command.indexOf(','));
        temp2 = Serial_Command.substring(Serial_Command.indexOf(',') + 1);
        temp1.toCharArray(carray, 10);
        CMD_Steps = atol(carray);
        temp2.toCharArray(carray, 10);
        Target_SPS = atol(carray);
        digitalWrite(LeftWheelDir,L_FWD);
        digitalWrite(RightWheelDir,R_FWD);
        End_Move_Alert = 1;
        Steps = 0;
        Target_Steps = CMD_Steps;
        Steps_To_Go = CMD_Steps;
        ACC_Flag = 1;
        Command = 0;
        //response to Pi
        //Serial.println(Command);
        //Serial.println(CMD_Steps);
        //Serial.println(Target_SPS);
      }
      else if (Serial_Command.startsWith("R")) // REVERSE MOTION command
      {
        temp1 = Serial_Command.substring(1, Serial_Command.indexOf(','));
        temp2 = Serial_Command.substring(Serial_Command.indexOf(',') + 1);
        temp1.toCharArray(carray, 10);
        CMD_Steps = atol(carray);
        temp2.toCharArray(carray, 10);
        Target_SPS = atol(carray);
        digitalWrite(LeftWheelDir,L_REV);
        digitalWrite(RightWheelDir,R_REV);
        End_Move_Alert = 1;
        Steps = 0;
        Target_Steps = CMD_Steps;
        Steps_To_Go = CMD_Steps;
        ACC_Flag = 1;
        Command = 1;
        //response to Pi
        Serial.println(Command);
        Serial.println(CMD_Steps);
        Serial.println(Target_SPS);
      }
      else 
      {
        if (!Serial_Command.equalsIgnoreCase("")) 
        {
          //Serial.println("BAD");
        }

      }

      Serial_Command = "";
    }
  }
    Command_String = Command_String

What does this accomplish?

only relevant portions

In whose opinion?

Do the same issues exist if the data is sent to the Arduino from the Serial Monitor app?

This Python - Arduino demo may be helpful.

It may need a little modification for Python 3 as it was written with 2.7

...R

Paul, the Command_String = Command_String is an artifact leftover from a variable name change - obviously, does nothing.

Regarding "whos opinion", if I included the whole program, it would exceed the 9000 character limit so I would say "Not relevant"

No, the behaviour is different when entering data in through the serial monitor, that worked perfectly.

Robin2, I will review that link, I don't recall seeing that one in all my Google searches - hopefully and answer is hiding in there!

Thanks

No, the behaviour is different when entering data in through the serial monitor, that worked perfectly.

The reason that I ask is to determine which side of the communications link has the problem. If the Arduino works properly when getting data from a different source, that strongly suggests that the problem isn't on the Arduino end.

Hi Paul S,

Yes, I agree, I think it is a problem with the Python output from the PI, but honestly, I am not sure of anything.

I have done a lot of serial interfacing over the years but never ran into so much trouble doing something so simple as this.

Thanks again!

Chris D

Hi again forum folks,

I am now able to communicate to the UNO, it echos back the data it received. This “round trip” of data is good about 30 to 40% of the time. I still cannot see where or how the data is getting messed up. There seems to be no pattern (like every 5th time) that I can see. I tried with and without the Arduino monitor running, with and without the Arduino plugged into the PC, wiggling the wires didn’t affect anything, different values to transmit, removed the line feed and CR, etc.

I am very much at a loss as to what is going on here. Sure would appreciated any help.

Chris D

Arduino Code…

// Serial interface variables
String Serial_Command; // String input from command prompt
String temp1, temp2; // temporary strings
char inByte; // Byte input from command prompt
char carray[10]; // character array for string to int // manipulation

//--------------------------------------------------------------------------------------------------------------
// Start up code
void setup()
{
  //configure / activate serial port
  Serial.begin(9600);           // set up Serial library at 9600 bps
}

//--------------------------------------------------------------------------------------------------------------
void loop()
{
  
  // Fetch commands from serial port if there are any
  if (Serial.available() > 0) {
    inByte = Serial.read();
      Serial.write(inByte); // echo back to pi everything that was sent in this command

    // only input if a letter, number, =,?,+,< ,> are typed!
    if ((inByte >= 65 && inByte <= 90) || (inByte >= 97 && inByte <= 122) || (inByte >= 48 &&     inByte <= 57) || inByte == 44 || inByte == 60|| inByte == 61|| inByte == 62 || inByte == 63) {
      Serial_Command.concat(inByte);
    }// end serial.available
    // Process command when NL/CR are entered:

    // Handle the incomming commands from the Raspberry Pi
    if (inByte == 62) { // key on > character as end of command
      //Serial.println("");
      inByte = 0;
      // Respond to a command... more code would be here to process command
      Serial_Command = "";
    }
  }
}// END main loop

Python 3.4 code…

# This routine will send a command to the Arduino and echos it back


from time import sleep
import serial
err_cnt = 0

#Start up code to open the serial port
ser = serial.Serial("/dev/ttyS0",9600, timeout =1)
print("Hello, waiting for data...")
if ser.isOpen():
    print(ser.name, "is open")
    print(ser)

def Send_Command_To_Pi2(Command_String):
    global err_cnt
    ser.flushInput()
    ser.flushOutput()
    ser.write(Command_String.encode('ascii'))
    sleep(.05)
    Buffer = ser.inWaiting()
    print("Buffer ",Buffer)
    try:
        indata = bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())     
        indata = indata + bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())
        indata = indata + bytes.decode(ser.read())
        return indata
    except:
        err_cnt += 1
        return "ERROR"


while True:
    print("------------------------")
    L_Steps = 12345
    S_Steps = str(L_Steps)
    L_SPS  = 3500
    S_SPS = str(L_SPS)   
    Command_String = "<F" + S_Steps + "," + S_SPS + ">"
    Return_Val = Send_Command_To_Pi2(Command_String)
    print("S ",Command_String)
    print("R ",Return_Val) 
    sleep(1)
    if ((inByte >= 65 && inByte <= 90) || (inByte >= 97 && inByte <= 122) || (inByte >= 48 &&     inByte <= 57) || inByte == 44 || inByte == 60|| inByte == 61|| inByte == 62 || inByte == 63) {

That would be far easier to understand as:

    if ((inByte >= 'A' && inByte <= 'Z') ||
        (inByte >= 'a' && inByte <= 'z') ||
        (inByte >= '0' && inByte <= '9) ||
         inByte == ',' || inByte == '<' ||
         inByte == '=' || inByte == '>' ||
         inByte == '?')
    {

Your python code assumes that all the data that is available to read is a complete packet, nothing more and nothing less. That is, as you are seeing, a piss-poor assumption.

There IS a reason for sending start and end of packet markers.

PaulS, I do not profess to be a C expert, nor a Arduino expert, nor a Python expert. I’m only trying to learn how to do what i want to do and only ask for help after countless hours of trying and researching via Google.

While I see that your change to the code is line-by-line as opposed to all on one line, I don’t see the difference or how it will fix the problem I am asking about.

As for start and end packet markers, The < is the start marker and the > is the end marker. I might not be parsing that correctly yet, and that could be the problem - I just don’t know so therefore I come to the forum for support and answers.

Chris D

While I see that your change to the code is line-by-line as opposed to all on one line, I don’t see the difference or how it will fix the problem I am asking about.

You don’t see the difference between having to look up what 97 means vs. having it right there in the code? Hmmm…

As for start and end packet markers, The < is the start marker and the > is the end marker.

When you send data TO the Arduino, yes. When you send data FROM the Arduino, you send back exactly what you received, including the start and end markers. But, on the python side, you just read whatever is available at some point in time, and assume that it is a complete packet, and nothing more than one packet. I want to know why the start and end markers are important going TO the Arduino but not coming FROM the Arduino.

At this point in development, I have just (since this morning) been able to get data from the Pi to the Arduino and back again somewhat reliably. Prior to that, I was only able to send data to the UNO and get jiberish back from the UNO after the first communication.

Having data come back that nearly matches, on a regular basis, is a huge step forward from where I started based on all the available information I have found in Google searches. In fact, the vast majority of examples out there wouldn't even function as they were written for Python 2.7 (without any indication what version they were written for).

So, to answer your question, at this point in testing and development, I don't care what order the data comes back to the Pi, only that it comes back as valid data, not random data like I am getting. If the characters are shifted by one or two places, it doesn't matter at this time. What matters is getting consistent data that is repeatable.

Lets forget for a moment that there are starting and ending characters and that the data is just a constant stream. Wouldn't it make sense that what I send is what I should get back?

Chris D.

Chris_D:
Lets forget for a moment that there are starting and ending characters and that the data is just a constant stream. Wouldn't it make sense that what I send is what I should get back?

I don't understand the logic of your thinking.

You are not sending a continuous stream of data. In your Original Post you said you want to send "F25000,3500". An end-marker is an essential part of identifying a message like that and using start- and end-markers makes it more reliable.

If you want to send back what you receive then you first need to receive it correctly - like the code in my link in Reply #2 does.

Also have a look at Serial Input Basics which may explain things a little more clearly. The same techniques are applicable for a PC program.

...R

My thinking is this....

In order to divide and conquer this problem, I have broken down the programs even more. I send one byte to the UNO and I get one byte back from it. I then compare the two, and if they are different it is a communications failure (corrupt data). I do this repeatedly in a loop.

This eliminates everything else in the conversation we have had about having start or end markers, eliminates proper formatting, eliminates as must as possible (that I can think of) to just send and receive data. All of that can be worked out AFTER I can get a byte from the PI to the UNO and back again reliably.

At this point of boiling down as many variables as I can, I have determined that if I use serial.write(complete string with encoding) from python, it appears it overflows the buffer on the UNO causing corrupt data.

If instead I send out 1 byte at a time using serial.write and pause .01 seconds between, my exchange of data is far more reliable. In previous attempts writing the whole command (F123456,6500) in one write statement, I was lucky if I could get 5 exchanges without an error of some sort. Sending one byte at a time has yielded exchanges in excess of 2500 bytes without an error or corrupt data.

While sending one by at a time, one of the most common corruptions is F being sent but f being returned. I find that very unusual.

Until I can figure out how to send and receive very basic data between these two Micros, nothing else matters in the program and what I need to do with the processing of that information. So, my focus is to figure out how to get the reliable exchange of data and deal with the parsing etc. later.

I hope that makes sense.

Chris D

Chris_D:
My thinking is this....

....

I hope that makes sense.

Not really - when there is already a working example available to you.

If there is something in the examples you don't understand then tell me and I will try to explain.

...R

Hi Robin, I suspect you are not aware that your example program does not run in Python 3.4?

At the moment, I cannot recall the error as I am at work now, but it has to do with the serial commands and I believe something to do with the buffer etc.

I did the basic Python 2.x to 3.x mods like the print statements, but couldn’t get past that issue.

This is the same problem I have run into with many other serial examples I found and tried.

If I didn’t already have the vast majority of my software written in 3.4, I would have started with 2.x to begin with.

While continueing testing of data echange, I did find something very useful and telling.

Earlier I mentioned that a common issue had to do with sending an F and getting an f in return? Well, that happens with almost all letters and some other characters.

The most reliably exchange characters are 0 ~ 9 < > and |.

I believe the problem has to do with the encode statements I am using on the Python side. If memory serves me, using the encode was the solution to one example program that wouldn’t work (2.x).

Chris D

Chris D

Chris_D:
Hi Robin, I suspect you are not aware that your example program does not run in Python 3.4?

I acknowldeged that possibility in Reply #2

I daresay it will be a lot easier to modify it for Python 3 than the reinvent the whole thing from the ground up. Alternatively, write a new Python 3 program that follows the same principles.

I have not yet experimented with Python 3. What messages do you get when you try to run my program without modifications?

...R

Hi Robin,

I agree completely, use what is there rather than write new! If it weren't for the Python 3 issues, my robot would have been happily running around my house a few weeks ago.

As I am at work now, I can't run the program to see what the errors are until tomorrow morning. I will report what I find at that time.

Thanks again!

Chris D

Chris_D:
If it weren't for the Python 3 issues, my robot would have been happily running around my house a few weeks ago.

Best reason I can think of to use Python 2.7 :slight_smile:

When you post the error messages I will have a look.

...R

The following Python version works with Python 3.4.3.

It was a real PITA figuring out how to get an ascii character into a string. I have marked the changes

# 19 July 2014
# 08 Dec 2016 - updated for Python3

# in case any of this upsets Python purists it has been converted from an equivalent JRuby program

# this is designed to work with ... ArduinoPC2.ino ...

# the purpose of this program and the associated Arduino program is to demonstrate a system for sending 
#   and receiving data between a PC and an Arduino.

# The key functions are:
#    sendToArduino(str) which sends the given string to the Arduino. The string may 
#                       contain characters with any of the values 0 to 255
#
#    recvFromArduino()  which returns an array. 
#                         The first element contains the number of bytes that the Arduino said it included in
#                             message. This can be used to check that the full message was received.
#                         The second element contains the message as a string


# the overall process followed by the demo program is as follows
#   open the serial connection to the Arduino - which causes the Arduino to reset
#   wait for a message from the Arduino to give it time to reset
#   loop through a series of test messages
#      send a message and display it on the PC screen
#      wait for a reply and display it on the PC

# to facilitate debugging the Arduino code this program interprets any message from the Arduino
#    with the message length set to 0 as a debug message which is displayed on the PC screen

# the message to be sent to the Arduino starts with < and ends with >
#    the message content comprises a string, an integer and a float
#    the numbers are sent as their ascii equivalents
#    for example <LED1,200,0.2>
#    this means set the flash interval for LED1 to 200 millisecs
#      and move the servo to 20% of its range

# receiving a message from the Arduino involves
#    waiting until the startMarker is detected
#    saving all subsequent bytes until the end marker is detected

# NOTES
#       this program does not include any timeouts to deal with delays in communication
#
#       for simplicity the program does NOT search for the comm port - the user must modify the
#         code to include the correct reference.
#         search for the lines 
#               serPort = "/dev/ttyS80"
#               baudRate = 9600
#               ser = serial.Serial(serPort, baudRate)
#


#=====================================

#  Function Definitions

#=====================================

def sendToArduino(sendStr):
    ser.write(sendStr.encode('utf-8')) # change for Python3


#======================================

def recvFromArduino():
    global startMarker, endMarker
    
    ck = ""
    x = "z" # any value that is not an end- or startMarker
    byteCount = -1 # to allow for the fact that the last increment will be one too many
    
    # wait for the start character
    while  ord(x) != startMarker: 
        x = ser.read()
    
    # save data until the end marker is found
    while ord(x) != endMarker:
        if ord(x) != startMarker:
            ck = ck + x.decode("utf-8") # change for Python3
            byteCount += 1
        x = ser.read()
    
    return(ck)


#============================

def waitForArduino():

    # wait until the Arduino sends 'Arduino Ready' - allows time for Arduino reset
    # it also ensures that any bytes left over from a previous message are discarded
    
    global startMarker, endMarker
    
    msg = ""
    while msg.find("Arduino is ready") == -1:

        while ser.inWaiting() == 0:
            pass
        
        msg = recvFromArduino()

        print (msg) # python3 requires parenthesis
        print ()
        
#======================================

def runTest(td):
    numLoops = len(td)
    waitingForReply = False

    n = 0
    while n < numLoops:
        teststr = td[n]

        if waitingForReply == False:
            sendToArduino(teststr)
            print ("Sent from PC -- LOOP NUM " + str(n) + " TEST STR " + teststr)
            waitingForReply = True

        if waitingForReply == True:

            while ser.inWaiting() == 0:
                pass
            
            dataRecvd = recvFromArduino()
            print ("Reply Received  " + dataRecvd)
            n += 1
            waitingForReply = False

            print ("===========")

        time.sleep(5)


#======================================

# THE DEMO PROGRAM STARTS HERE

#======================================

import serial
import time

print ()
print ()

# NOTE the user must ensure that the serial port and baudrate are correct
# serPort = "/dev/ttyS80"
serPort = "/dev/ttyACM0"
baudRate = 9600
ser = serial.Serial(serPort, baudRate)
print ("Serial port " + serPort + " opened  Baudrate " + str(baudRate))


startMarker = 60
endMarker = 62


waitForArduino()


testData = []
testData.append("<LED1,200,0.2>")
testData.append("<LED1,800,0.7>")
testData.append("<LED2,800,0.5>")
testData.append("<LED2,200,0.2>")
testData.append("<LED1,200,0.7>")

runTest(testData)


ser.close

…R

Hi Robin,

The Arduino code is untouched thus far.

The Python code, I changed....

All print statements as such FROM print msg TO print(msg) etc.
The serial port statement serPort = "/dev/ttyS0" to fit the Raspberry Pi

Run programs, on PI getting this error...

TypeError: Can't convert 'bytes' object to str implicitly.

on line 79 "ck = ck + x"

That is far as I got so far.

Chris D