optimizing pyserial and arduino serial communication

Projects Interfacing w/ Software on the Computer
1

I'm doing research sketches for an upcoming project that will use a RaspberryPi as a master and some form of Arduino as an LED pixel strip control. The RPI will also be controlling eight servos via a i2c breakout and four seperate channels of sound generation. It will be a busy Pi. I've chosen to write the master program on the Pi in Python.

My current stage of learning is setting up the Pi/Arduino serial interface. I'm writing the Python on my Mac and communicating using pyserial to a Mega 2560 via USB. The Arduino sketch is set up to divide a 150 pixel strip into three sections which can be controlled individually or in parallel. It takes a comma separated list of two fields; the first identifying which section of the strip, the second identifying the position on the strip. This code works great when I am just giving commands via the Serial Monitor.

In the Python I'm just running a for loop to send out simple position information. When I send out a '0,position', which runs all three sections in paralell it works fairly well, with only a little jitter and rarely skipping and LED. When I start to send separate sections commands, '1,position''2,position''3,position' it starts to get really jittery and will go totally haywire after a few loops.

I'm asking for advice on how to optimize the communication between the two devices. I know I'm missing something in the timing, because sometimes it runs smoother when I'm having Python print it's internal position marker to the shell.

Here's the Arduino code:

#include <Adafruit_NeoPixel.h>

#define PIN 2
#define numPix 150


// Parameter 1 = number of pixels in strip
// Parameter 2 = Arduino pin number (most are valid)
// Parameter 3 = pixel type flags, add together as needed:
//   NEO_KHZ800  800 KHz bitstream (most NeoPixel products w/WS2812 LEDs)
//   NEO_KHZ400  400 KHz (classic 'v1' (not v2) FLORA pixels, WS2811 drivers)
//   NEO_GRB     Pixels are wired for GRB bitstream (most NeoPixel products)
//   NEO_RGB     Pixels are wired for RGB bitstream (v1 FLORA pixels, not v2)
Adafruit_NeoPixel strip = Adafruit_NeoPixel(numPix, PIN, NEO_RGB + NEO_KHZ800);

int positions[3];
int index;

void setup(){
  
  Serial.begin(9600);
  strip.begin();
  strip.show();
  
}

void loop(){

  
  if(Serial.available()>0)
  {

    //look for index
    index =Serial.parseInt();
   
    if (Serial.read()== ','){
       
      int tempPos=Serial.parseInt();
     
      switch (index)
      {
        //the parallel case
        case 0:
             positions[0]=tempPos;
             positions[1]=100-tempPos;
             positions[2]=100+tempPos;
              break;
         //the individual cases     
        case 1:
              positions[0]=tempPos;
              break;
        
        case 2:
              positions[1]=100-tempPos;
              break;
              
        case 3:
              positions[2]=100+tempPos;
              break;
                
      }//end switch
    
    }//end if serial.read = ','

  }//end serial ava

//this section turns a single pixel in each section on according to position data, and turns all others off
    
    for (int i = 0; i<numPix;i++){
      if(i==positions[0]||i==positions[1]||i==positions[2])
      {
            strip.setPixelColor(positions[0],255,255,255);
            strip.setPixelColor(positions[1],255,255,255);
            strip.setPixelColor(positions[2],255,255,255);
      }
      else if (i!=positions[0]&&i!=positions[1]&&i!=positions[2])
      {
        strip.setPixelColor(i,0,0,0);
      }
    }
           
    
    strip.show();
    delay(20);
    
   

}//end loop

And here is the simple Pyserial loop:

import serial,time;

ser = serial.Serial('/dev/tty.usbmodemfa131',9600)

ser.write('0,0')          ##my wakeup call to the Mega
print ser.readable()##its response
    
time.sleep(1)           ##everybody get ready
print "begin"

while 1:     ##go

    for i in range(0,50):

##the parallel section:

        ser.write('0'+','+str(i)+'\n')    ## this works the best
        
##the individual section with some variations on the ser.write
       
##        send1='1' +','+str(i)+'\n'
##        send2='2' +','+str(i)+'\n'
##        send3='3' +','+str(i)+'\n'
               
##        ser.write(send1+send2+send3)

###### or

##        ser.write(send1)
##
##        
##        ser.write(send2)
##
##        
##        ser.write(send3)
   
##        print send1+send2+send3

I'm not really a very good nuts and bolts programmer, I'm more a 'code by the seat of my pants holy s@%$ it works don't touch it' sort of guy. I realize with this project I'm going to have to dig a little deeper but I'm not sure where to start. Any advice you can give other than 'give up and run away' would be helpful. (We already signed the contract)

Mostly I'm guessing that it has to do with how I've got the timing set up in the Python, but I'm sure there's optimizations I don't know about in the Arduino too. Thanks for your time.

2

I think you need to follow a few strategies in parallel.

Most important is to reduce to the absolute minimum the number of bytes you transmit. That means sending numbers as bytes or integers and NOT as characters. That also means your Arduino code won't need to use parseInt().

Then you need to use the highest possible baud rate. The max for the Arduino IDE is 115200 but the Arduino can probably work up to 1,000,000.

You should organize your Arduino code so that the whole of the next command is available before you read any of it. In other words if a command contains 2 bytes use if (Serial.available >= 2) {. This is because the serial data is transmitted at a very slow speed by Arduino standards.

I would just read all the serial data into an array before making any attempt to interpret it.

Can you give a few examples of the sort of data you need to transmit to the Arduino?
Have you any estimate of how many bytes per second you need to transmit when the full project is working?

...R

3

I investigated sending floating point between Python/Arduino some time ago. Arduino does not natively support 64 bit doubles, So I mapped the 32 bit float and 64 bit double to certain level.

some code can be found here - Arduino Playground - IEEE754tools -

4

Well, I tried a few things hoping to have some progress before I came back, but while I've learned alot I'm not much further along. It seems that pyserial.write() only sends strings, not ints. So I'm guessing that I will have to do some sort of char conversion on the Arduino end. I'm re-reading the Arduino Cookbooks 'Serial Recieve Multiple Fields' sketch and seeing how I can convert it to my needs. I've fallen into the serial.parseInt() trap before...it's just so easy! I should have known it's a time hog.

Can you give a few examples of the sort of data you need to transmit to the Arduino?

Each machine is on an articulated platform. I'm going to take the position of the platform and use it to set the color of the LEDs on that particular platform, ie when it is all the way up, it will be white, when it is all the way down, red and gradations of the two in between. So all the data I need to send is which machine(which sections of the LED strip) and it's position(which color). I've arbitrarily chosen the range of 0 - 100 as my position information. So really, all I need to send is two numbers in the range 0-4 and 0-100. The Mega or maybe a Teensy with one of these PJRC Store will be doing all the heavy lifting from there. Each machine will have three strips of 150 pixels, 450 LEDs per machine for a total of 1800 pixels, though I'm considering slaving the three strips on each machine to the same data line for a 'code' total of 450 pixels.

Have you any estimate of how many bytes per second you need to transmit when the full project is working?

The machines don't move terribly fast, so I don't need an insane refresh rate, not doing video or anything close. I would like some animation like effects, but I'd make those functions on the Arduino side, maybe add another set of numbers to my serial send to trigger them, or just trigger them randomly. I know that at the number of pixels I'm pushing nothing is going to happen real fast anyway.

robtillaart, I looked at you library, and it gave me some insight into how I need to make my arrays.

The other pain in the butt is that I can't read the Arduino serial monitor while the python is running to see real time what it's getting, and I can't yet trust the pyserial.read() to give me any info I can use.

Thanks

5

The other pain in the butt is that I can't read the Arduino serial monitor while the python is running to see real time what it's getting, and I can't yet trust the pyserial.read() to give me any info I can use.

h an LCD)

You could add a Serial display in parallel to the Arduino TX line to see what is send
(or a 2nd Arduino with an LCD)

6

leith00000:
It seems that pyserial.write() only sends strings, not ints.

As far as I can see write() sends bytes which can have any sort of data in them - it's up to you what you put into them. A string could contain 2 bytes that together represent an integer.

When I asked for

a few examples of the sort of data

I was being very literal. I just want to see what the data looks like - for example is it "A1234B7638" or something else entirely.

Regarding the bytes per second question, do you think you need 10, 100 or 1000?

...R

7

In pyserial, whenever I tried to send an int I got a traceback of "TypeError: 'int' object is not iterable." I just tried converting to both a binary and a hex, and those do work (ser.write(bin(integer))). Right now I'm looking over how easy it would be to convert the hex on the arduino, because then I can send all the info I need in a byte. Serial.write() returns how many bytes it took to send a message and 100 being hex64, it fits nicely. This one might do the trick...Arduino-Hex-Decimal-Conversion/hex_dec.ino at master · benrugg/Arduino-Hex-Decimal-Conversion · GitHub

I'm not sure what you mean on further examples of the data. At the Arduino I really just need a comma seperated list of two integers...machine one position 55 would just be 1,55; machine two position 100 would be 2,100.

As far as bytes per second...this is just positional information, not the actual refresh of the LED's, so if I can get this hex encoding to work, I'll need a byte per machine, and I don't think I'll need to update more than twice a second, at four machines that's eight bytes a second, so 10 bytes a second seems like a nice round number.

8

Oops...my math is wrong because I misunderstood what serial.write() returns...it's not bits, its bytes. Four bytes to send a two digit hex. That makes eight bytes a machine twice a second, sixteen bytes a second minimum.

9

I think you are confused between an integer as something that Python understands and the bytes of data that constitute the in-memory representation of the integer. It is the latter that is sent with write().

In Arduino terms an integer value of 4112 is stored in memory as 0001 0000 0001 0000 which is the same as two adjacent bytes of 16 dec. So if you create a string with the values chr(16) chr(16) and send it using write() it can be directly interpreted by the Arduino as the int 4112. (minor errors excepted)

Another way to think of it is that 4112 = 16 * 256 + 16. The reverse of this will enable you to turn a number into the bytes that represent it.

You need to be aware that an int on a PC may not be the same as an int on an Arduino.

...R

10

So I've been working away at it.... I think I'm on the right track, but I'm not getting any response across Pyserial. It works fine when I'm using the Serial Monitor to input the data. I'm sending ascii values and re-assembling them with atoi on the arduino. The first digit designates which machine an is currently not in use, the next three set the brightness level of the strip. Here's my updated code. I've moved over to a Teensy3.1 instead of the Mega I was using before.

Python:

import serial,time;

ser = serial.Serial('/dev/tty.usbmodem484101',38400)
ser.write('0000000000000000')## wake up
time.sleep(1)

for j in range(1,5):
    for i in range(0,100):

 
        tempnum=(j*1000)+i   ##make a four digit number in the form that I need it

        print tempnum
        buff= str(tempnum)+'\n'  ##make it into a string, add a newline
 
        
        

        for h in range(0,5):
          
            print ord(buff[h]) ##to monitor what is sending in the Python shell
            ser.write(buff[h])   ##send out each character in series
 


 

    
    time.sleep(.5)

Arduino:

#include <OctoWS2811.h>

unsigned char bytecount;
char incomingByte[4];

const int ledsPerStrip = 150;

DMAMEM int displayMemory[ledsPerStrip*6];
int drawingMemory[ledsPerStrip*6];

const int config = WS2811_GRB | WS2811_800kHz;

OctoWS2811 leds(ledsPerStrip, displayMemory, drawingMemory, config);

void setup(){
  pinMode(1, OUTPUT);   //is this necessary?....I just lifted it from the example code
  digitalWrite(1, HIGH);
  digitalWrite(1, LOW);
  leds.begin();

  for(int i = 0;i<ledsPerStrip;i++)//set everything to off
    {
      leds.setPixel(i,0,0,0);
      leds.show();
    }

  Serial.begin(38400);

}//end setup

void loop(){

 int result;

 while(Serial.available()&& bytecount < 4){
     
     if(Serial.available()){
       incomingByte[bytecount]=Serial.read();
       bytecount++;
       
     }

 }//end while

 result = atoi(&incomingByte[1]);    //I know nothing of C...this is spooky voodoo to me but it works

 for(int i = 0;i<ledsPerStrip;i++)
   {
     leds.setPixel(25,255,255,255);  //a test pixel so I know something is getting through to the Octo
     leds.setPixel(i,result,result,result); //this turns on everything at the level set by the three digits
     
   }
   leds.show();

 if (result>0&&bytecount>1){
   
   Serial.print(incomingByte[0]);
   Serial.print("  ");
   Serial.print(result);
   Serial.println("  result");
 }//end serial output

      
   result=0;
   bytecount=0;
   Serial.flush();  //I don't know if this is necessary or if it's even doing anything
  

  
}
11

Was it something I said?

12

leith00000:
Was it something I said?

What?

I've just seen your post but I don't have time to study it now. I will try to look at it this evening or in the morning.

You might look at the Python code in this demo if you are just interested in sending characters. I had assumed you were able to do that.

...R

13

Thanks...I'm sorry, no offense intended...I'm just considering giving up code and taking up self trepanation. I've gotten it to work finally with a delay(500) in the Arduino code. I think I'm having timing issues. I also tried replacing atoi with a result  =((incomingByte[1])-48)*100+((incomingByte[2])-48)*10+((incomingByte[3])-48); At this point I'm ready to live with lag.

14

Here is another demo that shows various ways of interpreting data.

I think you should either start with my working demo or strip everything out of your Arduino code except the stuff connected with getting the Serial data and work on that until it works. You should start by sending the data from the Arduino IDE so you can use Serial.println() to see the results. Then write a Pythong program to send the same stuff as you were typing in the Serial Monitor. If you can write your code so that it flashes an LED when the correct stuff is received you will get feedback even when you can't use the Serial Monitor.

When you have figured out Python you can make it receive and display the stuff that would appear in the Serial Monitor so you can debug your Arduino code while connected to Python.

...R

15

Ah, back to first principles so to speak. You're right, I was hoping to get lucky....plug in the right libraries and away we go. I'll post when I've got something. Thanks for your time.

16

At least the motor control test went ok....