How to send binary data with arduino and read it with pyserial

Im trying to figure out how to send and receive binary data using arduino to send the data and python to receive it I am able to run my program right now using Serial.print() and Serial.println() just fine. Except I need to give it a performance boost.

I am sending over data like so:
Serial.write(pot0holder);
Serial.write(".");
Serial.write(pot1holder);
Serial.write(".");
Serial.write(i);
Serial.write(",");
Serial.write(pot0holder+30);
Serial.write(".");
Serial.write(pot1holder+30);
Serial.write(".");
Serial.write(i);
Serial.write(",");
Serial.write(pot0holder+60);
Serial.write(".");
Serial.write(pot1holder+60);
Serial.write(".");
Serial.write(i);
Serial.write(",");
Serial.write(pot0holder+90);
Serial.write(".");
Serial.write(pot1holder+90);
Serial.write(".");
Serial.write(i);
Serial.write("\n");

Where pot0holder and pot1holder are just analog values and i is an integer that is incremented with each succesive series of prints.

I have yet to be able to read this with my python script in a way such that I can convert it into a string to be used for other operations in my Python program.

Any help is greatly appreciated.

What is the purpose of sending a value and then sending it again, with a constant added to it? What is the purpose of "i"?

I am just using I as a place holder for when I run my real simulation. I is the cycle that the program is currently on it will change at roughly the same rate but it's just a place holder for the same type of data being sent over.

This is wrong:

      Serial.write(".");

it must be:

      Serial.write('.');

Your "integer" i can never exceed 255.
Your "potholder" values can also not exceed 255.

Are you expecting it to be human readable or no?

I am reading a pressure sensor so the analog values can go up to 1023 and then I am sending those values to a python gui running on tkinter. I just want to be able to have the original analog values. on the python side

I just want to be able to have the original analog values. on the python side

Sending 0 in binary takes two bytes. Sending 0 in ASCII takes 2 (including a delimiter).

Sending 10 in binary takes two bytes. Sending 10 in ASCII takes 3.

Sending 100 in binary takes two bytes. Sending 100 in ASCII takes 4.

Sending 1023 in binary takes two bytes. Sending 1023 in ASCII takes 5.

Given that the number of bytes in ASCII is never more than 2.5 times the number in binary, and sending the value in ASCII is far simpler, why do you (think you) want to send the data in binary?

I need to increase the speed at which i am able to read data in using pyserial after looking through the forums I saw a few posts saying the fastest way to receive data would be by sending binary data. I currently use Serial.print() to send the data and I am unable to read it fast enough

You never answered the first question in reply #1.

Please post your entire working sketch (the one that sends in ASCII). It is probably the only hope of getting help with this. Use code tags. The code tags make the code look

like this

when posting source code files. It makes it easier to read, and can be copied with a single mouse click. Also, if you don't do it, some of the character sequences in the code can be misinterpred by the forum code as italics or funny emoticons. Use a standard indentation to clearly show the code blocks. Never put more than one statement per line. Place any brackets by themselves on a separate line. Use blank lines sparingly, no more than one at a time. Before posting the code, use Ctrl-T in the IDE to reformat the code in a standard format, which makes it easier for us to read.

The reason I am asking for this, is that it is easier to see what you need to accomplish from reading the code, than it is from asking you 200 questions.

For reading binary data in Python you probably need to use the unpack() function. You would not normally have delimiters BETWEEN items of binary data - just at the start and the end.

This Python Binary Data demo may be of interest.

...R

int analog0 = 0;
int analog1 = 1;

int sensor0;
int sensor1;

int pot0holder;
int pot1holder;
String Forceholder;

unsigned long i = 0;

boolean Sensordata = false;
 
const byte numChars = 32;
char receivedChars[numChars];
boolean newData = false;

unsigned long CurrentMillis = 0;
unsigned long PrintMillis = 0;
int PrintValMillis = 50;
unsigned long SensorMillis = 0;
int SensorValMillis = 50;

void setup() {
  Serial.begin(250000);
}

void loop()
{
  CurrentMillis = millis();
  recvWithStartEndMarkers();
  commands();
  sensordata();
}

void sensordata()
{
  if (CurrentMillis - SensorMillis >= SensorValMillis)
  {
    sensor0 = analogRead(analog0);
    pot0holder = sensor0;
    sensor1 = analogRead(analog1);
    pot1holder = sensor1;
    i += 1;
    Serial.print(pot0holder);
    Serial.print(".");
    Serial.print(pot1holder);
    Serial.print(".");
    Serial.print(i);
    Serial.print(",");
    Serial.print(pot0holder+30);
    Serial.print(".");
    Serial.print(pot1holder+30);
    Serial.print(".");
    Serial.print(i);
    Serial.print(",");
    Serial.print(pot0holder+60);
    Serial.print(".");
    Serial.print(pot1holder+60);
    Serial.print(".");  
    Serial.print(i);
    Serial.print(",");
    Serial.print(pot0holder+90);
    Serial.print(".");
    Serial.print(pot1holder+90);
    Serial.print(".");
    Serial.println(i);
    Serial.println(potcolumn);
    SensorMillis += SensorValMillis;
   }
}

void recvWithStartEndMarkers()
{
    static boolean recvInProgress = false; //creates variable visible to only one function with boolean
    static byte ndx = 0;
    char startMarker = '<'; //sets begin condition
    char endMarker = '>'; //sets end condition
    char rc; //sets variable type to char

    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read(); //sets rc equal to serial value

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }
        else if (rc == startMarker) {
            recvInProgress = true;
        }
    }
}

void commands()
{
  if (newData == true)
  {
    if (receivedChars[0] == 'T')
    {
      PrintValMillis = atoi(&receivedChars[1]); //atoi -> Converting strings to integer
    }
    else if (receivedChars[0] == 'L')
    {
      i = 0;
    }
  }
  newData = false;
}

Above is the code that I am using on an uno to prototype my design. All this really does is produce the data and every time i send it it resets the i variable to zero this is really just for my python GUI which I'll include in another post as it won't fit onthis one.

import Tkinter
import serial
from matplotlib.backends.backend_tkagg import FigureCanvasTkAgg
from matplotlib.figure import Figure
from matplotlib import pyplot as plt
import matplotlib.animation as animation
from collections import deque
import random
import time

class App:
    def __init__(self, master):
        self.arduinoData = serial.Serial('com5', 250000, timeout=None)

        frame = Tkinter.Frame(master)

        self.running = False
        self.ani = None

        self.run = Tkinter.LabelFrame(frame, text="Testing", borderwidth=10, relief=Tkinter.GROOVE, padx=10, pady=10)
        self.run.grid(row=0, column=0, padx=20, pady=20)

        self.run_respiration = Tkinter.Button(self.run, text="RUN",bd=10, height=5, width=10, command=self.getData)
        self.run_respiration.grid(row=0, column=0, padx=5, pady=5)

        self.test_options = Tkinter.LabelFrame(frame, text="Test Options", borderwidth=10, relief=Tkinter.GROOVE, padx=10, pady=10 )
        self.test_options.grid(row=0, column=1, padx=20, pady=20)

        self.stop = Tkinter.Button(self.test_options, text="STOP", bd=10, height=5, width=10, command=self.stopTest)
        self.stop.grid(row=0, column=0, padx=5, pady=5)


        self.fig = plt.Figure()
        self.ax1 = self.fig.add_subplot(211)
        self.line0, = self.ax1.plot([], [], lw=2)
        self.line1, = self.ax1.plot([], [], lw=2)
        self.line2, = self.ax1.plot([], [], lw=2)
        self.line3, = self.ax1.plot([], [], lw=2)
        self.ax2 = self.fig.add_subplot(212)
        self.line4, = self.ax2.plot([], [], lw=2)
        self.line5, = self.ax2.plot([], [], lw=2)
        self.line6, = self.ax2.plot([], [], lw=2)
        self.line7, = self.ax2.plot([], [], lw=2)
        self.canvas = FigureCanvasTkAgg(self.fig,master=master)
        self.canvas.show()
        self.canvas.get_tk_widget().grid(row=0, column=4, padx=20, pady=20)
        frame.grid(row=0, column=0, padx=20, pady=20)

    def getData(self):
        if self.ani is None:
            self.k = 0
            self.arduinoData.flushInput()
            self.arduinoData.write("<L>")
            return self.start()
        else:
            self.arduinoData.write("<L>")
            self.arduinoData.flushInput()
            self.ani.event_source.start()
        self.running = not self.running

    def stopTest(self):
        self.arduinoData.write("<H>")
        if self.running:
            self.ani.event_source.stop()
        self.running = not self.running

    def start(self):
        self.xdata = []
        self.pressure1 = []
        self.displacement1 = []
        self.cycle1 = []
        self.pressure2 = []
        self.displacement2 = []
        self.cycle2 = []
        self.pressure3 = []
        self.displacement3 = []
        self.cycle3 = []
        self.pressure4 = []
        self.displacement4 = []
        self.cycle4 = []
        self.k = 0
        self.limit = 300
        self.arduinoData.flushInput()
        self.timer()
        self.ani = animation.FuncAnimation(
            self.fig,
            self.setData,
            interval=10,
            repeat=True)
        self.arduinoData.write("<L>")
        self.running = True
        self.ani._start()


    def readData(self):
        if (self.arduinoData.inWaiting()>0):
            self.xdata.append(self.k)
            #t = time.time()
            x = self.arduinoData.readline()
            #print time.time()-t
            strip_data = x.strip()
            split_data = x.split("|")
            actuator1 = split_data[0].split(".")
            actuator2 = split_data[1].split(".")
            actuator3 = split_data[2].split(".")
            actuator4 = split_data[3].split(".")
            self.pressure1.append(int(actuator1[0]))
            self.displacement1.append(int(actuator1[1]))
            self.cycle1 = int(actuator1[2])
            self.pressure2.append(int(actuator2[0]))
            self.displacement2.append(int(actuator2[1]))
            self.cycle2 = int(actuator2[2])
            self.pressure3.append(int(actuator3[0]))
            self.displacement3.append(int(actuator3[1]))
            self.cycle3 = int(actuator3[2])
            self.pressure4.append(int(actuator4[0]))
            self.displacement4.append(int(actuator4[1]))
            self.cycle4 = int(actuator4[2])
            if self.k > 0:
                self.line0.set_data(self.xdata, self.pressure1)
                self.line1.set_data(self.xdata, self.pressure2)
                self.line2.set_data(self.xdata, self.pressure3)
                self.line3.set_data(self.xdata, self.pressure4)
                self.line4.set_data(self.xdata, self.displacement1)
                self.line5.set_data(self.xdata, self.displacement2)
                self.line6.set_data(self.xdata, self.displacement3)
                self.line7.set_data(self.xdata, self.displacement4)
                if self.k < 49:
                    #self.ax1.set_ylim(min(self.pressure1)-1, max(self.pressure4) + 1)
                    self.ax1.set_ylim(0, 1000)
                    self.ax1.set_xlim(0, self.k+1)
                    #self.ax2.set_ylim(min(self.displacement1)-1, max(self.displacement4) + 1)
                    self.ax2.set_ylim(0, 1000)
                    self.ax2.set_xlim(0, self.k+1)
                elif self.k >= 49:
                    #self.ax1.set_ylim(min(self.pressure1[self.k-49:self.k])-1, max(self.pressure4[self.k-49:self.k]) + 1)
                    self.ax1.set_ylim(0, 1000)
                    self.ax1.set_xlim(self.xdata[self.k-49], self.xdata[self.k-1])
                    #self.ax2.set_ylim(min(self.displacement1[self.k-49:self.k])-1, max(self.displacement4[self.k-49:self.k]) + 1)
                    self.ax2.set_ylim(0, 1000)
                    self.ax2.set_xlim(self.xdata[self.k-49], self.xdata[self.k-1])
                if self.cycle1 >= self.limit:
                    self.running = False
                    self.ani = None
                    self.ani.event_source.stop()
                    self.running = not self.running
            self.k += 1


    def setData(self, i):
            return self.line0, self.line1, self.line2, self.line3, self.line4, self.line5, self.line6, self.line7,



    def timer(self):
        self.readData()
        root.after(1, self.timer)

root = Tkinter.Tk()
app = App(root)
root.mainloop()

This is just a simple gui that reads data and plots it with the matplotlib function. The data backs up in the serial buffer because it's unable to read, graph, and update in under .05s the major bottle neck appears to be the readline() function when I profiled the code. I've tried reading in blocks of data, running everyhthing on separate timers but The readline takes too long. I think the best way to get around this would be to have the reading in a background process but I don't have the experience to write that yet. I've been trying to work in the multiprocessing library to do this but have been unsuccessful

Sorry I thought I answered reply #1 with my reply #2.

"i" is just a placeholder for what I will be sending in my real program. "i" will be incremented whenever an event occurs it will go up to 60,000, and there will be 3 more variables like "i" so to simulate this I am just reusing "i", because it's really similar to if i were to send "i","x","y","z" all incremented by 1 each cycle. So in short for brevity

If you want help with Python code create a simple version that does not need TKinter as all that stuff just confuses things.

Get the communication with the Arduino working before bothering with the GUI stuff.

...R

emg184:
Sorry I thought I answered reply #1 with my reply #2.

"i" is just a placeholder for what I will be sending in my real program. "i" will be incremented whenever an event occurs it will go up to 60,000, and there will be 3 more variables like "i" so to simulate this I am just reusing "i", because it's really similar to if i were to send "i","x","y","z" all incremented by 1 each cycle. So in short for brevity

No, that is an answer to the second sentence.

It seems horribly redundant to send a number, and then the same number with constants added to it. Can't you just add those at the receiving end?

Sorry should've explained that better

emg184:
Sorry should've explained that better

Well, I'm all ears.

aarg:
Well, I'm all ears.

Not me. I'm all thumbs.

Opposable, I hope.

I am able to run the communication between the two fast enough when I am performing no other tasks that was the first thing I did like you suggested it's just that when I add all the other tasks in is when I am unable to read fast enough. The biggest bottle neck was the reading time for the incoming data. So I am looking into reducing the read time which from what I've seen on the forums seems to be done by sending binary rather than ascii data.

emg184:
I am able to run the communication between the two fast enough when I am performing no other tasks that was the first thing I did like you suggested it's just that when I add all the other tasks in is when I am unable to read fast enough. The biggest bottle neck was the reading time for the incoming data. So I am looking into reducing the read time which from what I've seen on the forums seems to be done by sending binary rather than ascii data.

Post the program that works but is not fast enough. It will give us a better understanding of what you are trying to achieve.

...R