Serial.available > 0 from a raspberry pi

I'm working on a rover that eventually will be autonomous, I have a GPS and compass module connected to a Arduino mega, the ESC and servos are controlled by the mega through a PCA9685 servo board. The mega communicates to a raspberry pi via the serial port. I have a transmitter with 2 joysticks connected to a uno, it sends the joystick references to the rover via xbee wi-fi modules. All that works fine.

Now what I'm trying to do is send the speed and direction references to the pi and just send them back to the mega. Once I know I can send the references up and back I'll use the pi for navigation. Using python in the pi I can send the references up, my problem is sending them back and it seems to be a problem with using Serial.available in the mega.

To read the serial port I use the following

while(!Serial){}
if(Serial.available() >0){
PiSpdRef = Serial.parseInt();
PiDircRef = Serial.parseInt();
}

I have found that if I use Serial.available() > 0 in the if statement it doesn't read the serial port. If I change that to 1 so the if statement is always true it does read the data but I get inconsistent numbers, I assume because it's not reading the data correctly, port timing is off or something like that. I do have a 0.1 second delay in the python script between read and write of the data.

What am I doing wrong with the serial.available reference?

Thanks
John

This Simple Python - Arduino demo may help.

...R

the while statement is not necessary (i've never used it). try removing it (or at least putting it in setup()) and post your entire code

@Robin2 you have helped me many times before and obviously I had a similar question a few years ago as I use this method to read the data I am sending from the transmitter, there is a second function, parseData(), that splits the 3 variables I am sending into individual variables. I send the data to Serial1 on the mega.

I have a book on using a pi and Arduino and they use the method I show in my post. Although their example works I couldn't incorporate it into my main sketch. Several times I have said I should do it the same way as yours, sometimes I should listen to the voice in my head.

I will work on using your method today and let you know how I make out.

Thanks
John

@gcjr, I have tried it without the while statement, I figured that would just tie up processing of the rest of my sketch. I didn't post the whole sketch as it is to large to include. I will try Robin2's method and see how it goes.

thanks for you suggestion
John

why don't you using that approach in a smaller program that just reads and prints what is read

@gcjr a great idea. will be easier to debug before I put it into my main sketch

gcjr:
the while statement is not necessary (i've never used it). try removing it (or at least putting it in setup()) and post your entire code

The while statement is only needed when the processor has a native USB port, such as the atmega32u4 used on the Leonardo, Micro, etc, because it takes a significant amount of time to establish the USB connection and you can lose some of the serial data. I know from experience that a Serial.print() immediately after Serial.begin() on a Leonardo will never show up on the serial monitor.

Is this a more general issue of cross platform serial communication ? Do parameters like baud rate etc. match ?

This description of integrating Arduino and Pi over serial looks quite good. Raspberry Pi Arduino Serial Communication - Everything You Need To Know - The Robotics Back-End

Your using Serial, the programing port, on the Mega to connect to the Pi ?

What happens if you use, Serial1, Serial2 or Serial3 on the Mega to talk to the Pi ?

I use the serial port as I can just plug a cable into the USB on both the pi and the mega. I call up the Arduino sketch in the desktop on the pi and upload it from there

The mega communicates to a raspberry pi via the serial port

i think what srnet is saying is that the serial ports of the esp32 and mega are connected to the PC, not each other

@Robin2, your example works great, I've printed both sides out so I can study them more and understand how it works.

Thanks again for your help
John

@Robin2, I'm sure I'm missing the obvious here but after a couple of days of trying various things I'm getting no where and would greatly appreciate you reviewing what I have and letting me know what the problem is.

I copied the Python and Arduino examples you gave me the link to, they work fine. I'm now trying to incorporate the Arduino code into my sketch. My sketch reads two joysticks and a switch from a transmitter, with an Uno, via xBee wi-fi. That part works fine. Ultimately I want to send that data to the Pi and send other data back but for now I'm just trying to get it to work as you have it. I use your "recvWithStartEndMarkers()" to decode the string from the transmitter. So as the "recvWithStartEndMarkers()" on the Python communications wouldn't interact I've changed the name of it and all the variables that are used. Not sure if they would of but one of the things I tried to isolate the problem.

I made those changes in the copy of your code and it also works fine, but when I add that to my code I do not get a response from the Python code. I do get the first three messages, setting up the port, waiting for the Arduino and that the Arduino is ready, but nothing after that. I've commented out the code that I use to read the transmitter and just have the calls to the 2 functions for receiving and replying to the Python and it just doesn't work. I've made no changes to the Python code. I couldn't add the Python code as the message was to large.

Here's the copy of your Arduino sketch with the names changed

// This is very similar to Example 3 - Receive with start- and end-markers
//    in Serial Input Basics   http://forum.arduino.cc/index.php?topic=396450.0

const byte numChars_pi = 64;
char receivedChars_pi[numChars_pi];

boolean newData_pi = false;

byte ledPin = 13;   // the onboard LED

//===============

void setup() {
    Serial.begin(115200);

    pinMode(ledPin, OUTPUT);
    digitalWrite(ledPin, HIGH);
    delay(200);
    digitalWrite(ledPin, LOW);
    delay(200);
    digitalWrite(ledPin, HIGH);

    Serial.println("<Arduino is ready>");
}

//===============

void loop() {
    recvFromPi();
    replyToPython();
}

//===============

void recvFromPi() {
    static boolean recvInProgress_pi = false;
    static byte ndx = 0;
    char startMarker_pi = '<';
    char endMarker_pi = '>';
    char rc_pi;

    while (Serial.available() > 0 && newData_pi == false) {
        rc_pi = Serial.read();

        if (recvInProgress_pi == true) {
            if (rc_pi != endMarker_pi) {
                receivedChars_pi[ndx] = rc_pi;
                ndx++;
                if (ndx >= numChars_pi) {
                    ndx = numChars_pi - 1;
                }
            }
            else {
                receivedChars_pi[ndx] = '\0'; // terminate the string
                recvInProgress_pi = false;
                ndx = 0;
                newData_pi = true;
            }
        }

        else if (rc_pi == startMarker_pi) {
            recvInProgress_pi = true;
        }
    }
}

//===============

void replyToPython() {
    if (newData_pi == true) {
        Serial.print("<This just in ... ");
        Serial.print(receivedChars_pi);
        Serial.print("   ");
        Serial.print(millis());
        Serial.print('>');
            // change the state of the LED everytime a reply is sent
        digitalWrite(ledPin, ! digitalRead(ledPin));
        newData_pi = false;
    }
}

//===============

And here's my sketch

// Test communication to and from pi

// temporary array for use when parsing
const byte numChars = 64;
char receivedChars[numChars];
char tempChars[numChars];

// variables to hold the parsed data
int Throttle_Ref = 0;
int Steering_Ref = 0;
int servoRef_F = 500; //position refererence for front servo
int servoTrim_F = 15; // trim to center front servo
int servoRef_R = 500; //position refererence for rear servo
int servoTrim_R = 0; // trim to center rear servo
int escRef = 500; // ESC speed cpontrol reference
int escTrim = 0; // ESC speed control trim
int ThrottleOutValue = 0;
int SteeringOutValue = 0;
unsigned long Comm_Time = 0;
unsigned long Comm_Time_2 = 0;
int Comm_Delay_Time = 1000;
int index = 0;
int Control_Mode = 0;
boolean newData = false;
boolean newDataPi = false;
int PiSpdRef = 0;
int PiDircRef = 0;
int PiSum = 0;

// Variables to send data to Xmitter
char a;
char startMarker = '<';
char endMarker = '>';
char comma = ',';
String SendBuffer;
unsigned long StartTime;
int DelayTime;

// data to pi
  String Hdng_pi;   // Compass Heading yp pi
  String Lat_pi;    // Latitude to pi
  String Lon_pi;    // Longitude to pi
  String Sw_pi;     // Switch status to pi
  String Data_pi;   // Combined values to pi
  String SpdRef_pi; // Speed reference to pi
  String Direc_pi;  // Direction to pi
  String Sum_pi;

// for comm to/from pi

const byte numChars_pi = 64;
char receivedChars_pi[numChars_pi];
boolean newData_pi = false;
  
void setup() {

 Serial.begin(115200); // Serial Monitor
 Serial1.begin(9600);  // Read Data from Wifi Controller
 Serial.println("<Arduino is Ready>");

}

void loop() {

  // read transmitter
/* 
 recvWithStartEndMarkers(); // Read data from controller
  if (newData == true) {
    strcpy(tempChars, receivedChars);
    // this temporary copy is necessary to protect the original data
    //   because strtok() used in parseData() replaces the commas with \0

    parseData(); // Split data into separate variables
    newData = false;
  } //end if (newData == true)


 Serial.print(Throttle_Ref);
 Serial.print("\t");
 Serial.print(Steering_Ref);
 Serial.print("\t");
 Serial.println(Control_Mode);
*/
 // Comm to/from pi

// SpdRef_pi = Throttle_Ref;
// Direc_pi = Steering_Ref;
// PiSum = Throttle_Ref + Steering_Ref;
// Sum_pi = PiSpdRef + PiDircRef;

    recvFromPi();
    replyToPython();

} // end loop

void recvWithStartEndMarkers() { // reads variables from controller
  static boolean recvInProgress = false;
  static byte ndx = 0;
  //  char startMarker = '<';
  //  char endMarker = '>';
  char rc;

  while (Serial1.available() > 0 && newData == false) {
    rc = Serial1.read();

    if (recvInProgress == true) {
      if (rc != endMarker) {
        receivedChars[ndx] = rc;
        ndx++;
        if (ndx >= numChars) {
          ndx = numChars - 1;
        }
      }
      else {
        receivedChars[ndx] = '\0'; // terminate the string
        recvInProgress = false;
        ndx = 0;
        newData = true;
      }
    }

    else if (rc == startMarker) {
      recvInProgress = true;
    }
  } // end while (Serial1.available()

} // end void recvWithStartEndMarkers()

//============

void parseData() {      // split the data into its parts
  strcpy(tempChars, receivedChars);
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the string
  Throttle_Ref = atoi(strtokIndx);     // Throttle reference

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  Steering_Ref = atoi(strtokIndx);     // Steering reference

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  Control_Mode = atoi(strtokIndx);     // Watch Dog Communication Pulse

} // end void parseData()

//===============  Communications to pi

void recvFromPi() {
  
    static boolean recvInProgress_pi = false;
    static byte ndx = 0;
    char startMarker_pi = '<';
    char endMarker_pi = '>';
    char rc_pi;

    while (Serial.available() > 0 && newData_pi == false) {
        rc_pi = Serial.read();

        if (recvInProgress_pi == true) {
            if (rc_pi != endMarker_pi) {
                receivedChars_pi[ndx] = rc_pi;
                ndx++;
                if (ndx >= numChars_pi) {
                    ndx = numChars_pi - 1;
                }
            }
            else {
                receivedChars_pi[ndx] = '\0'; // terminate the string
                recvInProgress_pi = false;
                ndx = 0;
                newData_pi = true;
            }
        }

        else if (rc_pi == startMarker_pi) {
            recvInProgress_pi = true;
        }
    }
}

//===============

void replyToPython() {
  
    if (newData_pi == true) {
        Serial.print("<This just in ... ");
        Serial.print(receivedChars_pi);
        Serial.print("   ");
        Serial.print(millis());
        Serial.print('>');
        newData_pi = false;
    }
}

//===============

Thanks for your help and your time
John

Stumpy_L:
Here's the copy of your Arduino sketch with the names changed

Why have you changed the names - it is much easier to help when you use the code I am familiar with.

When everything works then you can change the names to make them meaningful for your project.

Also, please post an example of the message that is being sent by the Python program.

...R

Sorry but as mentioned I already had the function in my sketch of that name, that one reads the data from the transmitter on Serial1. I probably could use it for this as well but wanted to keep them as separate functions. I'll put them back and change the other one if that will be helpful.

in the meantime this is what I get in the shell as a response.

Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "copyright", "credits" or "license()" for more information.

============= RESTART: /home/pi/Arduino/Pi_Comm/Python_Comm_2.py =============
Serial port /dev/ttyACM0 opened Baudrate 115200
Waiting for Arduino to reset
Arduino is Ready

have restored your functions as they are in your example and have renamed the original function. I still have only the calls to your functions enabled and get the same results.

here is the restored revised sketch

// Test communication to and from pi

// temporary array for use when parsing
const byte numChars = 64;
char receivedChars[numChars];
char tempChars[numChars];

// variables to hold the parsed data
int Throttle_Ref = 0;
int Steering_Ref = 0;
int servoRef_F = 500; //position refererence for front servo
int servoTrim_F = 15; // trim to center front servo
int servoRef_R = 500; //position refererence for rear servo
int servoTrim_R = 0; // trim to center rear servo
int escRef = 500; // ESC speed cpontrol reference
int escTrim = 0; // ESC speed control trim
int ThrottleOutValue = 0;
int SteeringOutValue = 0;
unsigned long Comm_Time = 0;
unsigned long Comm_Time_2 = 0;
int Comm_Delay_Time = 1000;
int index = 0;
int Control_Mode = 0;
boolean newData_xmit = false;
boolean newData = false;
int PiSpdRef = 0;
int PiDircRef = 0;
int PiSum = 0;

// Variables to send data to Xmitter
char a;
char startMarker = '<';
char endMarker = '>';
char comma = ',';
String SendBuffer;
unsigned long StartTime;
int DelayTime;

// data to pi
  String Hdng_pi;   // Compass Heading yp pi
  String Lat_pi;    // Latitude to pi
  String Lon_pi;    // Longitude to pi
  String Sw_pi;     // Switch status to pi
  String Data_pi;   // Combined values to pi
  String SpdRef_pi; // Speed reference to pi
  String Direc_pi;  // Direction to pi
  String Sum_pi;

// for comm to/from pi

const byte numChars_pi = 64;
char receivedChars_pi[numChars_pi];
boolean newData_pi = false;
  
void setup() {

 Serial.begin(115200); // Serial Monitor
 Serial1.begin(9600);  // Read Data from Wifi Controller
 Serial.println("<Arduino is Ready>");

}

void loop() {

  // read transmitter
/* 
 recevFromXmitter(); // Read data from controller
  if (newData_xmit == true) {
    strcpy(tempChars, receivedChars);
    // this temporary copy is necessary to protect the original data
    //   because strtok() used in parseData() replaces the commas with \0

    parseData(); // Split data into separate variables
    newData_xmit = false;
  } //end if (newData == true)


 Serial.print(Throttle_Ref);
 Serial.print("\t");
 Serial.print(Steering_Ref);
 Serial.print("\t");
 Serial.println(Control_Mode);

 // Comm to/from pi

// SpdRef_pi = Throttle_Ref;
// Direc_pi = Steering_Ref;
// PiSum = Throttle_Ref + Steering_Ref;
// Sum_pi = PiSpdRef + PiDircRef;
*/
    recvWithStartEndMarkers();
    replyToPython();

} // end loop

void recevFromXmitter() { // reads variables from controller
  static boolean recvInProg_xmit = false;
  static byte ndx_xmit = 0;
  //  char startMarker = '<';
  //  char endMarker = '>';
  char rc_xmit;

  while (Serial1.available() > 0 && newData_xmit == false) {
    rc_xmit = Serial1.read();

    if (recvInProg_xmit == true) {
      if (rc_xmit != endMarker) {
        receivedChars[ndx_xmit] = rc_xmit;
        ndx_xmit++;
        if (ndx_xmit >= numChars) {
          ndx_xmit = numChars - 1;
        }
      }
      else {
        receivedChars[ndx_xmit] = '\0'; // terminate the string
        recvInProg_xmit = false;
        ndx_xmit = 0;
        newData_xmit = true;
      }
    }

    else if (rc_xmit == startMarker) {
      recvInProg_xmit = true;
    }
  } // end while (Serial1.available()

} // end void recevFromXmitter()

//============

void parseData() {      // split the data into its parts
  strcpy(tempChars, receivedChars);
  char * strtokIndx; // this is used by strtok() as an index

  strtokIndx = strtok(tempChars, ",");     // get the first part - the string
  Throttle_Ref = atoi(strtokIndx);     // Throttle reference

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  Steering_Ref = atoi(strtokIndx);     // Steering reference

  strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
  Control_Mode = atoi(strtokIndx);     // Watch Dog Communication Pulse

} // end void parseData()

//===============

void recvWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    char startMarker = '<';
    char endMarker = '>';
    char rc;

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

//===============

void replyToPython() {
    if (newData == true) {
        Serial.print("<This just in ... ");
        Serial.print(receivedChars);
        Serial.print("   ");
        Serial.print(millis());
        Serial.print('>');
            // change the state of the LED everytime a reply is sent
        //digitalWrite(ledPin, ! digitalRead(ledPin));
        newData = false;
    }
}

//===============

Stumpy_L:
I still have only the calls to your functions enabled and get the same results.

here is the restored revised sketch

You need to post the Python program also and explain what you mean by "same results"

In Reply #13 you said "I copied the Python and Arduino examples you gave me the link to, they work fine." So what have you changed?

...R

Here is the Arduino sketch and the Python code from your examples. When I load the Arduino sketch into the mega and use the Python code it works as it should. I then copied the functions from the Arduino sketch and pasted them in my sketch, attached on reply #16, call them in the loop as you do I get the following results on the python shell window

============= RESTART: /home/pi/Arduino/Pi_Comm/Python_Comm_2.py =============
Serial port /dev/ttyACM0 opened Baudrate 115200
Waiting for Arduino to reset
Arduino is Ready

I have changed nothing in your functions, I renamed my original functions that are used to read the transmitter plus I have commented out the calls to that function. So as far as I can see this should work the same as yours.

thanks

import serial
import time

startMarker = '<'
endMarker = '>'
dataStarted = False
dataBuf = ""
messageComplete = False

#========================
#========================
    # the functions

def setupSerial(baudRate, serialPortName):
    
    global  serialPort
    
    serialPort = serial.Serial(port= serialPortName, baudrate = baudRate, timeout=0, rtscts=True)

    print("Serial port " + serialPortName + " opened  Baudrate " + str(baudRate))

    waitForArduino()

#========================

def sendToArduino(stringToSend):
    
        # this adds the start- and end-markers before sending
    global startMarker, endMarker, serialPort
    
    stringWithMarkers = (startMarker)
    stringWithMarkers += stringToSend
    stringWithMarkers += (endMarker)

    serialPort.write(stringWithMarkers.encode('utf-8')) # encode needed for Python3


#==================

def recvLikeArduino():

    global startMarker, endMarker, serialPort, dataStarted, dataBuf, messageComplete

    if serialPort.inWaiting() > 0 and messageComplete == False:
        x = serialPort.read().decode("utf-8") # decode needed for Python3
        #data = (x.decode('utf-8').strip())
        #[SpdRef, Direc, Sum] = data.split(',')
        
        if dataStarted == True:
            if x != endMarker:
                dataBuf = dataBuf + x
            else:
                dataStarted = False
                messageComplete = True
        elif x == startMarker:
            dataBuf = ''
            dataStarted = True
    
    if (messageComplete == True):
        messageComplete = False
        return dataBuf
    else:
        return "XXX" 

#==================

def waitForArduino():

    # wait until the Arduino sends 'Arduino is ready' - allows time for Arduino reset
    # it also ensures that any bytes left over from a previous message are discarded
    
    print("Waiting for Arduino to reset")
     
    msg = ""
    while msg.find("Arduino is ready") == -1:
        msg = recvLikeArduino()
        if not (msg == 'XXX'): 
            print(msg)



#====================
#====================
    # the program


setupSerial(115200, "/dev/ttyACM0")
count = 0
prevTime = time.time()
while True:
            # check for a reply
    arduinoReply = recvLikeArduino()
    if not (arduinoReply == 'XXX'):
        print ("Time %s  Reply %s" %(time.time(), arduinoReply))
        
        # send a message at intervals
    if time.time() - prevTime > 1.0:
        sendToArduino("this is a test " + str(count))
        prevTime = time.time()
        count += 1
// This is very similar to Example 3 - Receive with start- and end-markers
//    in Serial Input Basics   http://forum.arduino.cc/index.php?topic=396450.0

const byte numChars_pi = 64;
char receivedChars_pi[numChars_pi];

boolean newData_pi = false;

byte ledPin = 13;   // the onboard LED

//===============

void setup() {
    Serial.begin(115200);

    pinMode(ledPin, OUTPUT);
    digitalWrite(ledPin, HIGH);
    delay(200);
    digitalWrite(ledPin, LOW);
    delay(200);
    digitalWrite(ledPin, HIGH);

    Serial.println("<Arduino is ready>");
}

//===============

void loop() {
    recvFromPi();
    replyToPython();
}

//===============

void recvFromPi() {
    static boolean recvInProgress_pi = false;
    static byte ndx = 0;
    char startMarker_pi = '<';
    char endMarker_pi = '>';
    char rc_pi;

    while (Serial.available() > 0 && newData_pi == false) {
        rc_pi = Serial.read();

        if (recvInProgress_pi == true) {
            if (rc_pi != endMarker_pi) {
                receivedChars_pi[ndx] = rc_pi;
                ndx++;
                if (ndx >= numChars_pi) {
                    ndx = numChars_pi - 1;
                }
            }
            else {
                receivedChars_pi[ndx] = '\0'; // terminate the string
                recvInProgress_pi = false;
                ndx = 0;
                newData_pi = true;
            }
        }

        else if (rc_pi == startMarker_pi) {
            recvInProgress_pi = true;
        }
    }
}

//===============

void replyToPython() {
    if (newData_pi == true) {
        Serial.print("<This just in ... ");
        Serial.print(receivedChars_pi);
        Serial.print("   ");
        Serial.print(millis());
        Serial.print('>');
            // change the state of the LED everytime a reply is sent
        digitalWrite(ledPin, ! digitalRead(ledPin));
        newData_pi = false;
    }
}

//===============

Stumpy_L:
Here is the Arduino sketch and the Python code from your examples. When I load the Arduino sketch into the mega and use the Python code it works as it should. I then copied the functions from the Arduino sketch and pasted them in my sketch, attached on reply #16, call them in the loop as you do I get the following results on the python shell window

That's a good bit clearer - thanks.

I have spent some time looking at your code but I can't see what the problem might be.

I suggest you start again with the unchanged programs from my Simple Python - Arduino demo. Get them working. Then add your code into those programs, testing after every change. Then it should become obvious what change is causing it to break.

Separately, the name changes for local variables WITHIN this function are a contradiction of the whole purpose of using functions and local variables. Local variables only have an existence within a function and their names cannot clash with names of variables used outside a function.

void recvFromPi() {
    static boolean recvInProgress_pi = false;
    static byte ndx = 0;
    char startMarker_pi = '<';
    char endMarker_pi = '>';
    char rc_pi;

You could have several similar functions - for example recvFromAnotherArduino() - with the exact same local variable names. Not changing the local variable names makes it much less likely that an error will creep in.

...R