Serial frame synchronization between Arduino and Python

nebulae · March 14, 2020, 8:14pm

I'm trying to log Arduino data with Python script in Raspberry via USB serial. The preferred data rate is quite high, 2.3kHz. The payload is a consistent 16 bytes of information.

I would like to use raw serial communication. But I'm not sure how to synchronize the frames to avoid framing errors. There is a lots of information on the internet. Unfortunately I still can't find an answer on how to implement them in practice. I certainly wouldn't invent a wheel. Please advise me what to look for.

The code to emulate the data source:

#include "FastCRC.h"

FastCRC16 CRC16;

unsigned long timer = 0;
long loopTime = 99999;   // microseconds

void timeSync(unsigned long deltaT)
{
  unsigned long currTime = micros();
  long timeToDelay = deltaT - (currTime - timer);
  if (timeToDelay > 5000)
  {
    delay(timeToDelay / 1000);
    delayMicroseconds(timeToDelay % 1000);
  }
  else if (timeToDelay > 0)
  {
    delayMicroseconds(timeToDelay);
  }
  else
  {
    // timeToDelay is negative so we start immediately
  }
  timer = currTime + timeToDelay;
}

float valueA = -999.9;
float valueB = 2.0;
float valueC = 3.0;
float valueD = 4.0;

void printDataWithCRC(float *data1, float *data2, float *data3, float *data4) {
  byte *byteData1 = (byte *)(data1);
  byte *byteData2 = (byte *)(data2);
  byte *byteData3 = (byte *)(data3);
  byte *byteData4 = (byte *)(data4);
  byte buf[16] = {
    byteData1[0], byteData1[1], byteData1[2], byteData1[3],
    byteData2[0], byteData2[1], byteData2[2], byteData2[3],
    byteData3[0], byteData3[1], byteData3[2], byteData3[3],
    byteData4[0], byteData4[1], byteData4[2], byteData4[3]
  };
  Serial.write(buf, 16);
  uint16_t crc = CRC16.ccitt(buf, sizeof(buf));
  Serial.write((byte) (crc >> 8));
  Serial.write((byte) crc);
}

double randomDouble(double minf, double maxf) {
  return minf + random(1UL << 31) * (minf - maxf) / (1UL << 31);
}

void setup() {
  Serial.begin(230400);
  timer = micros();
}

void loop() {
  timeSync(loopTime);
  //  float val1 = randomDouble(0.01, 9.99);
  //  float val2 = randomDouble(0.01, 9.99);
  //  float val3 = randomDouble(0.01, 9.99);
  //  float val4 = randomDouble(0.01, 9.99);
  float val1 = -999.9;
  float val2 = 1.0;
  float val3 = 5.5;
  float val4 = 99.9;
  printDataWithCRC(&val1, &val2, &val3, &val4);
}

Paul_KD7HB · March 14, 2020, 8:51pm

Are you going to use a common clock signal between the two boards?

Paul

nebulae · March 14, 2020, 9:00pm

Paul_KD7HB:
Are you going to use a common clock signal between the two boards?

Paul

Maybe you're asking that because I used the term "syncronization"? No. It'll be UART.

I presume it could be done with start/stop bit termination, but not sure how shall I do it. Or even better additionally detecting the idle between frames?

Robin2 · March 14, 2020, 10:01pm

Have a look at the examples in Serial Input Basics - simple reliable non-blocking ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

The technique in the 3rd example will be the most reliable. It is what I use for Arduino to Arduino and Arduino to PC communication. The start and end markers provide the synchronisation.

You can send data in a compatible format with code like this (or the equivalent in any other programming language)

Serial.print('<'); // start marker
Serial.print(value1);
Serial.print(','); // comma separator
Serial.print(value2);
Serial.println('>'); // end marker

Also have a look at this Simple Python - Arduino demo

...R

nebulae · March 15, 2020, 11:35am

Robin2:
Have a look at the examples in Serial Input Basics - simple reliable non-blocking ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

The technique in the 3rd example will be the most reliable. It is what I use for Arduino to Arduino and Arduino to PC communication. The start and end markers provide the synchronisation.

You can send data in a compatible format with code like this (or the equivalent in any other programming language)
Serial.print('<'); // start marker

Serial.print(value1);
Serial.print(','); // comma separator
Serial.print(value2);
Serial.println('>'); // end marker




Also have a look at this [Simple Python - Arduino demo](http://forum.arduino.cc/index.php?topic=598209.) 

...R

Thank you for the reply. Very helpful walkthrough. Although it is good for character comms, IMO start/end markers will trigger false termination conditions in binary. Perhaps you meant something else?

Robin2 · March 15, 2020, 12:10pm

nebulae:
IMO start/end markers will trigger false termination conditions in binary.

That's true. I did not know you are working with binary data.

First question is it essential to use binary data? Sending data as human readable text makes debugging much easier. I would only send binary data if the extra performance was essential.

Second question, have you the ability to make changes to the sending program so as to make the data (whether binary or text) easier to handle?

If you can change the sending program then there are at least two options. The simplest is to set aside two byte values (perhaps 254 and 255) as start and end markers. That will reduce the range of possible data values to 0 to 253 - a loss of resolution of less than 1%. If you can't do that then if you set aside three values as markers (say 253, 254 and 255) you can use 253 as a marker that says that the following byte is a data byte rather than a start-or end-marker. For example to send the data value 254 you would send 253 254 and the send the value 253 you would send 253 253

...R

nebulae · March 15, 2020, 12:59pm

Robin2:
First question is it essential to use binary data? Sending data as human readable text makes debugging much easier. I would only send binary data if the extra performance was essential.

The objective is to basically use the Arduino as a DAQ. I would like to be able to send 16 byte (24 in the future) time critical measurement data from Arduino to Raspberry at a relatively high speed of 2.3 kHz (or at least 1 kHz) and store it in a HDF5 file.

Robin2:
Second question, have you the ability to make changes to the sending program so as to make the data (whether binary or text) easier to handle?

Yes. It is an Arduino code.

Robin2:
If you can change the sending program then there are at least two options. The simplest is to set aside two byte values (perhaps 254 and 255) as start and end markers. That will reduce the range of possible data values to 0 to 253 - a loss of resolution of less than 1%. If you can't do that then if you set aside three values as markers (say 253, 254 and 255) you can use 253 as a marker that says that the following byte is a data byte rather than a start-or end-marker. For example to send the data value 254 you would send 253 254 and the send the value 253 you would send 253 253

Hrmm. It sounds like a very extraordinary solution. Similar to the one in your post "Demo of PC-Arduino comms using Python".

Robin2 · March 15, 2020, 1:46pm

nebulae:
Similar to the one in your post "Demo of PC-Arduino comms using Python".

That's hardly surprising - I had hoped it was identical, but maybe I overlooked some detail.

I would like to be able to send 16 byte (24 in the future) time critical measurement data from Arduino to Raspberry at a relatively high speed of 2.3 kHz (or at least 1 kHz)

That seems to mean that you want to send 24 x 2300 = 55200 bytes per second. Sending at 500,000 baud would fall a little short of that but at 1,000,000 baud there would be plenty of opportunity to include additional sync bytes.

If this was my project I would use either of the schemes that I suggested in Reply #5 to enable start- and end-markers to be added to the data.

It might also be a good idea to send some form of ID with each message so that the receiving program could identify if a message was missed. A single byte that increments from (say) 0 to 9 and then repeats would probably be sufficient.

...R

Power_Broker · March 15, 2020, 3:33pm

You can interface Python and Arduinos with binary data easily using the compatible libraries: pySerialTransfer and SerialTransfer.h.

pySerialTransfer is pip-installable and cross-platform compatible. SerialTransfer.h runs on the Arduino platform and can be installed through the Arduino IDE's Libraries Manager.

Both of these libraries have highly efficient and robust packetizing/parsing algorithms with easy to use APIs.

Example Python Script:

from time import sleep
from pySerialTransfer import pySerialTransfer as txfer

if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM13')
        
        link.open()
        sleep(2) # allow some time for the Arduino to completely reset
    
        link.txBuff[0] = 'h'
        link.txBuff[1] = 'i'
        link.txBuff[2] = '\n'
        
        link.send(3)
        
        while not link.available():
            if link.status < 0:
                print('ERROR: {}'.format(link.status))
            
        print('Response received:')
        
        response = ''
        for index in range(link.bytesRead):
            response += chr(link.rxBuff[index])
        
        print(response)
        link.close()
        
    except KeyboardInterrupt:
        link.close()

Example Arduino Sketch:

#include "SerialTransfer.h"

SerialTransfer myTransfer;

void setup()
{
  Serial.begin(115200);
  Serial1.begin(115200);
  myTransfer.begin(Serial1);
}

void loop()
{
  myTransfer.txBuff[0] = 'h';
  myTransfer.txBuff[1] = 'i';
  myTransfer.txBuff[2] = '\n';
  
  myTransfer.sendData(3);
  delay(100);

  if(myTransfer.available())
  {
    Serial.println("New Data");
    for(byte i = 0; i < myTransfer.bytesRead; i++)
      Serial.write(myTransfer.rxBuff[i]);
    Serial.println();
  }
  else if(myTransfer.status < 0)
  {
    Serial.print("ERROR: ");
    Serial.println(myTransfer.status);
  }
}

For more info, check out the Serial Input Advanced tutorial.

nebulae · March 15, 2020, 5:57pm

Power_Broker:
You can interface Python and Arduinos with binary data easily using the compatible libraries: pySerialTransfer and SerialTransfer.h.

pySerialTransfer is pip-installable and cross-platform compatible. SerialTransfer.h runs on the Arduino platform and can be installed through the Arduino IDE's Libraries Manager.

Both of these libraries have highly efficient and robust packetizing/parsing algorithms with easy to use APIs.

I went through your posts. It looks exactly what I need, all in one package.

Before it takes me understand how Python library works, I would like to ask for your help how to receive the data send from Arduino. I have my simple Arduino sketch to emulate the data source:

#include "SerialTransfer.h"

SerialTransfer myTransfer;

void sendToPC(float* data1, float* data2, float* data3, float* data4)
{
  byte* byteData1 = (byte*)(data1);
  byte* byteData2 = (byte*)(data2);
  byte* byteData3 = (byte*)(data3);
  byte* byteData4 = (byte*)(data3);
  byte buf[16] = {byteData1[0], byteData1[1], byteData1[2], byteData1[3],
                  byteData2[0], byteData2[1], byteData2[2], byteData2[3],
                  byteData3[0], byteData3[1], byteData3[2], byteData3[3],
                  byteData4[0], byteData4[1], byteData4[2], byteData4[3]
                 };
  for (uint8_t i = 0; i < 16; i++) {
    myTransfer.txBuff[i] = buf[i];
  }
  myTransfer.sendData(16);
}

void setup() {
  Serial.begin(115200);
}

void loop() {
  timeSync(loopTime);
  float val1 = -999.00;
  float val2 = 1.1234;
  float val3 = 5.0;
  float val4 = 9.99;
  sendToPC(&val1, &val2, &val3, &val4);
  delay(500);
}

Power_Broker · March 15, 2020, 8:15pm

nebulae:
Before it takes me understand how Python library works, I would like to ask for your help how to receive the data send from Arduino. I have my simple Arduino sketch to emulate the data source:

I'd be glad to help. Do you have a specific question?

Paul_KD7HB · March 15, 2020, 9:00pm

nebulae:
Maybe you're asking that because I used the term "syncronization"? No. It'll be UART.

I presume it could be done with start/stop bit termination, but not sure how shall I do it. Or even better additionally detecting the idle between frames?

I am confuser! Not the first or last time.

The standard for asynchronous data is for the BITS, 10 bits per 8 bits of data, to be SYNCHROUNOUS based on the bit rate, AKA Baud rate. The actual 10 bits are asynchronous because there is NO timing associated with them. Each set of 10 bits, start bit, 8 bits of data, stop bit, may have many seconds between each, or may have no delay time between them.

Are you attempting to only send 8 bits of data with nothing to indicate the starting of each set of bits? The old synchronous data transmissions were sent that way, with a set of known sync characters used to adjust the bit clock to match the incoming bits. In that case, no start bits or stop bits are necessary.

Paul

nebulae · March 15, 2020, 10:39pm

Power_Broker:
I'd be glad to help. Do you have a specific question?

Thanks! A Python example to receive the data from Arduino would be great. I'm sending a 16 byte frame for testing.

Paul_KD7HB:
I am confuser! Not the first or last time.

The standard for asynchronous data is for the BITS, 10 bits per 8 bits of data, to be SYNCHROUNOUS based on the bit rate, AKA Baud rate. The actual 10 bits are asynchronous because there is NO timing associated with them. Each set of 10 bits, start bit, 8 bits of data, stop bit, may have many seconds between each, or may have no delay time between them.

Are you attempting to only send 8 bits of data with nothing to indicate the starting of each set of bits? The old synchronous data transmissions were sent that way, with a set of known sync characters used to adjust the bit clock to match the incoming bits. In that case, no start bits or stop bits are necessary.

Paul

I probe Arduino's TX line with a logic analyzer and it shows me exactly 16 bytes of data when I "Serial.write()" 16 byte data. I guess it is 10 bits for text comms maybe?

Power_Broker · March 15, 2020, 11:26pm

nebulae:
Thanks! A Python example to receive the data from Arduino would be great. I'm sending a 16 byte frame for testing.

The built-in example already shows how to do this, but for a more tailored (and fully tested) example, see below:

Arduino Code:

#include "SerialTransfer.h"


SerialTransfer myTransfer;


void sendToPC(float data1, float data2, float data3, float data4)
{
  uint8_t sendLen = 0;
  
  myTransfer.txObj(data1, sizeof(data1), sendLen);
  sendLen += sizeof(data1);
  
  myTransfer.txObj(data2, sizeof(data2), sendLen);
  sendLen += sizeof(data2);
  
  myTransfer.txObj(data3, sizeof(data3), sendLen);
  sendLen += sizeof(data3);
  
  myTransfer.txObj(data4, sizeof(data4), sendLen);
  sendLen += sizeof(data4);
  
  myTransfer.sendData(sendLen);
}


void setup()
{
  Serial.begin(115200);
  myTransfer.begin(Serial);
}

void loop()
{
  float val1 = -999.00;
  float val2 = 1.1234;
  float val3 = 5.0;
  float val4 = 9.99;
  
  sendToPC(val1, val2, val3, val4);
}

Python Code:

import struct
from pySerialTransfer import pySerialTransfer as txfer


if __name__ == '__main__':
    try:
        link = txfer.SerialTransfer('COM17')
        link.open()
        
        while True:
            if link.available():
                data1 = struct.unpack('f', bytes(link.rxBuff[0:4]))[0]
                data2 = struct.unpack('f', bytes(link.rxBuff[4:8]))[0]
                data3 = struct.unpack('f', bytes(link.rxBuff[8:12]))[0]
                data4 = struct.unpack('f', bytes(link.rxBuff[12:16]))[0]
                
                print('data1: {}\ndata2: {}\ndata3: {}\ndata4: {}\n'.format(data1,
                                                                            data2,
                                                                            data3,
                                                                            data4))
            
            elif link.status < 0:
                    print('ERROR: {}'.format(link.status))
        
    except KeyboardInterrupt:
        link.close()

Paul_KD7HB · March 16, 2020, 1:44am

nebulae:
Thanks! A Python example to receive the data from Arduino would be great. I'm sending a 16 byte frame for testing.

I probe Arduino's TX line with a logic analyzer and it shows me exactly 16 bytes of data when I "Serial.write()" 16 byte data. I guess it is 10 bits for text comms maybe?

I bet your logic analyzer was told to look for async data and realized it needed to sync it's logic on a start bit and end with a stop bit, so just showed the actual data. Look at the same TX line with an oscilloscope.

Paul

nebulae · March 16, 2020, 12:38pm

Power_Broker:
The built-in example already shows how to do this, but for a more tailored (and fully tested) example, see below:

Many thanks for the tailored example! It works amazingly well. I wish it was the weekend. I could fully discover its potential:

Paul_KD7HB:
I bet your logic analyzer was told to look for async data and realized it needed to sync it's logic on a start bit and end with a stop bit, so just showed the actual data. Look at the same TX line with an oscilloscope.

Paul

Interesting to know it. I will try to check that.

Topic		Replies	Views
Arduino/Python Data Framing Library - Using flags, byte stuffing, and CRC16 Libraries	6	7531	May 6, 2021
Simple serial parsing Programming	11	3460	May 5, 2021
Poor serial performance when sending data from RaspberryPi to Arduino Interfacing w/ Software on the Computer	22	27086	May 6, 2021
Serial communication with Python - beginner qestion Interfacing w/ Software on the Computer	17	25975	November 13, 2022
Arduino serial communication with Raspberry Pi and Python3 Interfacing w/ Software on the Computer	10	8476	May 6, 2021

Serial frame synchronization between Arduino and Python

Related topics