Loosing data when reading over serial

In my project, I am trying to control an led strip with code from my computer. I do this by sending bytes using PySerial and then receiving it on the arduino. Sending two bytes at a time worked perfektly fine but starting with three, the data sometimes ends up in the wrong place. I am still trying to learn about Serial as I don't understand the inner workings with the buffering and everything well enough to find my problem. I have rewritten my code to be in line with the 'Serial Input Basics' by Robin2 to receive the bytes and I have created a minimum failing example on the python side.

import serial_helper as sh
import time

serialInst = sh.init_port(6)

time.sleep(5)

brightness = 253
speed = 0
bar = 50
start = 254
end = 255

for i in range(250):
    bar = 250 - i
    serialInst.write(start.to_bytes(1, "little"))
    serialInst.write(brightness.to_bytes(1, "little"))
    serialInst.write(bar.to_bytes(1, "little"))
    serialInst.write(i.to_bytes(1, "little"))
    serialInst.write(end.to_bytes(1, "little"))
    time.sleep(0.1)

This code worked fine for sending one or two bytes. I used 254 and 255 ans beginning and end because I try to get a large usable range and these two were the easiest to sacrifice. I don't know what would be best practice here but that would work for me.

#include <FastLED.h>

#define LED_PIN    6
#define LED_TYPE    WS2812B
#define COLOR_ORDER GRB

#define LED_COUNT 300

CRGB leds[LED_COUNT];

const byte recBytes = 32;
byte numReceived = 0;

boolean newData = false;

float t = 0;
float speed = 1;
int basscol = 200;
int bassbright = 255;
int midcol = 50;
int midbright = 150;
int buf;

const int BUFFER_SIZE = 5;
byte databuf[recBytes];
int numBytes = 0;

void setup() {
  databuf[1] = 0;
  Serial.begin(9600);

  delay(3000);
  
  FastLED.addLeds<LED_TYPE,LED_PIN,COLOR_ORDER>(leds, LED_COUNT).setCorrection(TypicalLEDStrip);
  FastLED.setBrightness(80); // Set BRIGHTNESS to about 1/5 (max = 255)
  Serial.setTimeout(5);
}

void loop() {
  recvBytesWithStartEndMarkers();
  if (newData) {
    bassbright = databuf[0];
    speed = 0.2+float(databuf[2])/100;
    //midcol = databuf[3];
    midbright = databuf[1];
    newData = false;
  }
  
  basscol++;

  empty(0, 299);
  colorRamp(150, 299, basscol, bassbright, 30, t);
  mirror(110, 149);
  //colorTrain(80, 149, midcol, midbright, 2);
  //mirror(150, 219);
  bar(0, midbright, 100, midcol, 100);
  
  FastLED.show();
  t += speed;
  delay(10);
  if (basscol > 255) {
    basscol -= 255;
  }
}

void recvBytesWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    int startMarker = 254;
    int endMarker = 255;
    byte rb;
   

    while (Serial.available() > 0) {
        rb = Serial.read();

        if (recvInProgress == true) {
            if (rb != endMarker) {
                databuf[ndx] = rb;
                ndx++;
                if (ndx >= recBytes) {
                    ndx = recBytes - 1;
                }
            }
            else {
                recvInProgress = false;
                numReceived = ndx;  // save the number for use when printing
                ndx = 0;
                newData = true;
            }
        }

        else if (rb == startMarker) {
            recvInProgress = true;
        }
    }
}

I used most of the read function from the example. The expected behavior would be a color ramp that gets faster and faster and a bar that drains with passing time. These features have worked all the time so there is no problem with the functions themselves. When I send three bytes, the bar mostly drains smoothly but sometimes jumps for one frame so some values are not read correctly. When I do a check and only set newData to True if numReceived == 3 then there are no spikes, but I am skipping data. I also tried to add a check if Serial.available() > 4 before the while so that I only read if there is enough data but that doesn't fix it either. Does anyone have an idea what could be wrong here? Any help is greatly appreciated.

Both baudrates are set to 9600, this is the code of serial helper:

import serial.tools.list_ports
import time

def init_port(port):
    ports = serial.tools.list_ports.comports()
    serialInst = serial.Serial()

    portsList = []

    for onePort in ports:
        portsList.append(str(onePort))

    for x in range(0, len(portsList)):
        if portsList[x].startswith("COM" + str(port)):
            portVar = "COM" + str(port)
            print(portVar)

    serialInst.baudrate = 9600
    serialInst.port = portVar
    serialInst.open()

    return serialInst

probably all you have to do is change that "while" to an "if"..
loop and read one byte at a time till you get the end, you potentially read too much using the while..

good luck.. ~q

Receiving Serial while driving the LEDs can be a problem. Interrupts are disabled while FastLED is sending data to the LEDs, and Serial needs an interrupt for each character it receives. With 300 LEDs, the interrupts are disabled for 9mS, which is nearly 50% of the time (the remainder of the code spends almost all its time in the 10mS delay), so there is a good chance you will lose some data.

The while is fine, its desirable to empty the receive buffer completely, or read until the end marker is received. Using an if instead would result in reading a maximum of one character each time loop executes, which in this code takes about 19mS.

really want to use that while with no transmission control, then maybe add a break once newData is true..
right now, the while is an issue..
sorry..
happy holidays.. ~q

You are correct, I had not noticed the OP left out the test for newData, the code from the Serial Input Basics page should be:

    while (Serial.available() > 0 && newData == false) {
1 Like

Aaah I see yeah if the LED writing disables receiving Serial then that makes sense that some data is lost. I took out the delay but the issue won't be fixed with that. And thank you for pointing out the error in the while loop I forgot to put it back in. So I think I understand the problem now but is there a way around that? I think I need about 20 fps for smooth animations.

The only thing I could imagine is synchronizing sender and receiver on a clock but that sounds complicated?

Another idea might be to send a request signal from the arduino and when the python script receives it it sends whatever data is available and then arduino waits for it to get back but then I might have the same problem on the python side right?

The workaround can be to only send a single byte from your python program and let the Arduino program echo it. The python program should only be allowed to send the next byte after it receives the echo.

The Arduino side for it is easy, I have no idea about the python side

    while (Serial.available() > 0) {
        rb = Serial.read();
        // echo
        Serial.write(rb);

No idea if you can achieve your framerate.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.