Problem parsing serial data

Using Arduino Programming Questions
1

I have a problem reading in serial data and assigning it to the FastLED array. Actually, it works fine most of the time but regularly, the number of bytes read by Serial.readBytesUntil() does not match what is expected (180). As you can see in the code below, I send the number of bytes read when it does not match 180 and I receive quite a few…

I end my messages with a \n (10). I tried extending the timeout but it does not help.

Ideas?

P.S. leds is the FastLED array.

void loop() {

  count = Serial.readBytesUntil(10, (char*)leds, TOTAL+1);
  
  if (count == TOTAL) {
    FastLED.show();
  } else {
    Serial.println(count);
  }

}
2

IDE 2.x is for Problems with the Arduino IDE itself NOT your project. It says so in the description of the section. Therefore I have moved your post here.

You might want to look at this How to get the best out of this forum before you proceed any further.

This doesn't make much sense. The count variable will just contain the last character received, and TOTAL+1 will do nothing to the total variable.

The

Serial.readBytesUntil(10

Will read the serial port until the character '10' is received, that is the hex value "0x0A" and is a carriage return you can get at the end of a String. note String and string are different things, the upper case S defines a totally different thing.

Also note that your serial input buffer is only 64 bytes long, so you might be sending them too fast and your buffer is overflowing. Try lowering the baud rate.

3

the challenge might be that you are actually sending binary data, not ASCII followed by the line feed '\n'

So whilst you wait for a line feed, it is very much possible that you also have 10 as part of the bytes you are receiving which will stop the Serial.readBytesUntil(character, buffer, length) command

it does I think (but is wrong).
He probably has TOTAL LEDS in his led strip and so want to get all the bytes plus the line feed terminator hence the + 1

Serial.readBytesUntil() reads characters from the serial buffer into an array. The function terminates (checks being done in this order) if the determined length has been read, if it times out (see Serial.setTimeout()), or if the terminator character is detected (in which case the function returns the characters up to the last character before the supplied terminator). The terminator itself is not returned in the buffer.

Serial.readBytesUntil() returns the number of characters read into the buffer. A 0 means that the lengthparameter <= 0, a time out occurred before any other input, or a termination character was found before any other input.

the leds array is probably a CRGB type so 4 bytes per pixel probably

2 Likes
5

IDE 2.x is for Problems with the Arduino IDE itself NOT your project. It says so in the description of the section. Therefore I have moved your post here.

Sorry about that. This breach of etiquette wasn't intentional. I'll be more careful next time.

This doesn't make much sense. The count variable will just contain the last character received, and TOTAL+1 will do nothing to the total variable.

Serial.readBytesUntil() actually returns the number of bytes read. The if was simply meant as a way to understand what was going on. TOTAL is the number of LEDs x 3 (because there are three colors per LED.

6

So whilst you wait for a line feed, it is very much possible that you also have 10 as part of the bytes you are receiving which will stop the Serial.readBytesUntil(character, buffer, length) command

Yeah, I now realize how stupid I am... I am sending a bunch of random bytes... surely one of them is bound to be 10. This has to be the issue. I'm going to check that and report back. Thanks for pointing out the obvious!

7

When in doubt read the manual.

You are not using this correctly.

8

When in doubt read the manual.

Thanks for taking time to help me out. It's very much appreciated. However, rest assured that I did read the manual, several times.

The problem, as @J-M-L (thanks!!!) swiftly pointed out, is that my separator (\n) has a byte value of 10 which is also used in the "regular" values I'm sending as part of the byte stream.

So, as a workaround, I am scaling my values to 0-254 and using 255 as the separator (I need a separator because other unrelated data will also need to be sent). However, it still does not seem to be detected. Here is the updated code (edited for clarity):

void loop() {
  int count = Serial.readBytesUntil(255, (char*)leds, NUM_LEDS * 3 + 1);
  Serial.println(count); // for debugging purposes
  FastLED.show();
}

leds is the FastLED array containing the LED values for R, G and B. Since I have 60 LEDs, it reads in 180 values plus the terminator. The code above works fine but...

If I supply 200 as the last parameter to readBytesUntil(), it returns 200 bytes. Shouldn't it return only the 180 bytes before the terminator?

This is the code on the other end that sends the bytes (Python):

def onOffToOn():

	for i in range(opColors.numSamples):

		r = int(opColors['r'][i]) # cast to int because those are floats
		v = int(opColors['g'][i])
		b = int(opColors['b'][i])

		opArduino.sendBytes(r.to_bytes(1, "big"), v.to_bytes(1, "big"), b.to_bytes(1, "big"))

	term = 255
	opArduino.sendBytes(int(term).to_bytes(1, "big"))

	return

Am I sending the 255 byte wrong? Why is my Arduino code not stopping to read bytes when it reaches the byte with a value of 255 ?

Thanks again!

9

I think the problem here is that you are only supplying us with snippets and not with complete code. This leaves too many unanswered questions.

Please supply full code for both ends and state what boards you are using, again for both ends. What sort of Python are you using?

10

I tried to isolate (and translate) the relevant bits to facilitate the readers' job but perhaps that did not help. Here is the full code for the receiving end (Arduino Uno connected to LED strip):

#include <FastLED.h>

const int BORNE = 6;
const int NOMBRE = 60;
unsigned long VITESSE = 115200;
const CRGB CORRECTION(255, 175, 235);
const int BRIGHTNESS = 32;
size_t count = 0;
CRGB dels[NOMBRE];

void setup() {

  Serial.begin(VITESSE);

  FastLED.addLeds<NEOPIXEL, BORNE>(dels, NOMBRE);
  FastLED.show();
  FastLED.setBrightness(BRIGHTNESS);
  FastLED.setCorrection(CORRECTION);

}

void loop() {
  count = Serial.readBytesUntil( 255, (char*)dels, NOMBRE * 3 + 1);
  if (count == NOMBRE * 3) FastLED.show();
}

The sending side is a Mac laptop running TouchDesigner. TouchDesigner runs this bit of code each time I want to update the LEDs:

opArduino = op("ser_arduino")
opCouleurs = op("nul_couleurs")

def onOffToOn(channel, sampleIndex, val, prev):

	for i in range(opCouleurs.numSamples):
		r = int(opCouleurs['r'][i])
		v = int(opCouleurs['g'][i])
		b = int(opCouleurs['b'][i])
		opArduino.sendBytes(r.to_bytes(1, "big"), v.to_bytes(1, "big"), b.to_bytes(1, "big"))

	opArduino.sendBytes(int(255).to_bytes(1, "big"))

	return

Hope this helps!

11

Is it important in this application that the bytes are received as fast as possible?

If no you can use code that works this way
your receive-function checks each and everybyte if this byte matches a start-character the commonly used character for this is "<"
if this character is detected the receiving of the RGB-data starts and goes on until an end-marker is received. In your case I would use indeed 255 to have 0-254 as brightness-values.

If speed really doesn't matters because you are not driving your RGB-leds dynamically fading in / fading out where each value of brightness is transmitted with the the datastream you could use ascii-coded digits.

In general if you describe not only your detail-problem but the overall application this information will help to find a good solution.

best regards Stefan

12

If you add a print statement on the Python side do you see 180 bytes of rgb data, or do you see a number of bytes greater than 200?

In my opinion the Arduino code looks pretty good, and I suspect the Python sending side.

13

I don't know TouchDesigner. Does this reboot the arduino? does it happen only once and never close the Serial line thereafter?

what do the parameters 1 and big mean in r.to_bytes(1, "big")

14

The first thing I notice is that this does not match the snippets you posted earlier.

void loop() {

  count = Serial.readBytesUntil(10, (char*)leds, TOTAL+1);
  
  if (count == TOTAL) {
    FastLED.show();
  } else {
    Serial.println(count);
  }

}

As against

void loop() {
  count = Serial.readBytesUntil( 255, (char*)dels, NOMBRE * 3 + 1);
  if (count == NOMBRE * 3) FastLED.show();
}

I know you changed the delimiter to 255 but what is all the

NOMBRE * 3 + 1

About?

And you had us believe you were sending 180 bytes?

You answered:-

So what is TouchDesigner? I have searched the Apple App Store and come up with nothing. Where did you get this app from? What Mac Hardware and OS are you using to run it on?

The name TouchDesigner implies some sort of touch screen, something not found on many Macs.

If people want to run your applications to try and find out what is wrong then they need to know this information.

15

NOMBRE is the number of leds in the strip

presumably the sender code, sends 3 bytes for each leds with

opArduino.sendBytes(r.to_bytes(1, "big"), v.to_bytes(1, "big"), b.to_bytes(1, "big"))

so that's 3 * NOMBRE bytes

and then sends one end marker (255)

opArduino.sendBytes(int(255).to_bytes(1, "big"))

so the total number bytes sent is 3 * NOMBRE + 1

as Serial.readBytesUntil() returns the number of characters read into the buffer and that the terminator itself is not returned in the buffer, if the frame has been received in full, the Serial.readBytesUntil call will return NOMBRE * 3 hence the

if (count == NOMBRE * 3) FastLED.show();

———

of course this all makes sense if what is sent from the undocumented TouchDesigner matches the expectations... so that's where I would dig first.. are we receiving the expected input?

16

Yes me too, but it is difficult with the lack of information we are receiving.

17

Indeed

@djip we need more info to help you :innocent:

are you using this TouchDesigner?

seems indeed they use python to extend the possibilities

18

TouchDesigner is a node-based visual programming environment used to create interactive installations, control live performances and manipulate media in realtime. It can be enhanced with custom Python programming.

In this specific case, I generate visual textures that are then sent over serial to LED strips (via Arduino). Actually, this is for a class I teach where students are expected to work with LED strips in realtime.

Is it important in this application that the bytes are received as fast as possible?

Yes. It needs to be fast because the texture sent to the strips are constantly changing. When I tried parsing each character, I was getting much slower results. By reading the serial data directly into the FastLED buffer, it yields a very significant speed improvement.

Just to be extra clear, the code I have works (since I corrected the mistake pointed out by @J-M-L).

The only problem that remains is that I want to introduce a separator in the byte stream to append other control information after the LED colour information. When I use readSerialBytes(), it returns more bytes than expected. I send 180 bytes (60 LEDs with 3 colours each) and then a separator. However, it is as though readSerialBytes() does not recognize the separator and just continues reading data. I just do not understand why the function does not stop at my separator (255).

what do the parameters 1 and big mean in r.to_bytes(1, "big")

This converts the red value (an int) to a byte using Big Endian.

If people want to run your applications to try and find out what is wrong then they need to know this information.

I'm afraid it would be difficult to replicate the whole setup on your side. That's why I'm trying to narrow it down to something that's not too involved for you guys.

Here is the code with translated variable names (see comments in code):

#include <FastLED.h> // https://fastled.io/

const int LED_PIN = 6;
const int NUM_LEDS = 60;
unsigned long SPEED = 115200;
const CRGB CORRECTION(255, 175, 235);
const int BRIGHTNESS = 32;
size_t count = 0;
CRGB leds[NUM_LEDS];

void setup() {
  Serial.begin(SPEED);

  FastLED.addLeds<NEOPIXEL, LED_PIN>(dels, NUM_LEDS);
  FastLED.show();
  FastLED.setBrightness(BRIGHTNESS);
  FastLED.setCorrection(CORRECTION);
}

void loop() {

  // This works fine:
  count = Serial.readBytesUntil(255, (char*)dels, NUM_LEDS * 3 + 1);

  // However, if I do this (below), "count" will be greater than the 
  // expected 180. It is as though my separator byte (255) is not being 
  // recognized. 
  count = Serial.readBytesUntil(255, (char*)dels, 200);
}

So, I guess I'm not sending the separator character correctly but I have no idea why it does not work. In Python, this is the code that adds the separator at the end of the colour bytes:

opArduino.sendBytes(int(255).to_bytes(1, "big"))

I have no clue why the 255 byte generated in this way is not recognized by the Serial.readBytesUntil() method. Its basically the same as when I send the colours and the colours are received okay.

I hope this is clearer. Thanks all for your help.

19

can you try something a bit different ?

#include <FastLED.h> // https://fastled.io/

const int LED_PIN = 6;
const int NUM_LEDS = 60;
unsigned long SPEED = 115200;
const CRGB CORRECTION(255, 175, 235);
const int BRIGHTNESS = 32;
size_t count = 0;
CRGB leds[NUM_LEDS];

void setup() {
  Serial.begin(SPEED);

  FastLED.addLeds<NEOPIXEL, LED_PIN>(leds, NUM_LEDS);
  FastLED.show();
  FastLED.setBrightness(BRIGHTNESS);
  FastLED.setCorrection(CORRECTION);
}

void loop() {

  if (Serial.peek() != 255) {     // wait until the current frame is done as we don't know where we were in the sending
    Serial.read();                // get rid of the end marker
    size_t n = 0;
    byte rgb = 0;
    while (n < NUM_LEDS) {
      if (Serial.peek() != -1) { // a new byte is ready
        leds[n][rgb++] = Serial.read();
        if (rgb >= 3) {
          rgb = 0;
          n++;
        }
      }
    }
    FastLED.show();
  }
}

it's not very robust but at least there is no risk from timeout in reading the data

20

Thanks for the example. However, it doesn't seem to be working. I'm not sure why we would get rid of the first byte with Serial.read(). The first byte is part of the data. We need to get rid of the 255 byte, don't we?

In any case, I tried something like this earlier on and it was quite a bit slower...

I guess I'm just going to stick with Serial.readBytes() instead of using Serial.readBytesUntil().

21

Oops my bad of course it's == 255; not != 255

#include <FastLED.h> // https://fastled.io/

const int LED_PIN = 6;
const int NUM_LEDS = 60;
unsigned long SPEED = 115200;
const CRGB CORRECTION(255, 175, 235);
const int BRIGHTNESS = 32;
size_t count = 0;
CRGB leds[NUM_LEDS];

void setup() {
  Serial.begin(SPEED);

  FastLED.addLeds<NEOPIXEL, LED_PIN>(leds, NUM_LEDS);
  FastLED.show();
  FastLED.setBrightness(BRIGHTNESS);
  FastLED.setCorrection(CORRECTION);
}

void loop() {

  if (Serial.read() == 255) {     // wait until the current frame is done as we don't know where we were in the sending
    size_t n = 0;
    byte rgb = 0;
    while (n < NUM_LEDS) {
      if (Serial.peek() != -1) { // a new byte is ready
        leds[n][rgb++] = Serial.read();
        if (rgb >= 3) {
          rgb = 0;
          n++;
        }
      }
    }
    FastLED.show();
  }
}

so basically until we see 255 we ignore anything - that's on the assumption there is already an incoming frame and we missed the start

once we see 255, we ignore that one and then we know we are at the start of a new frame, so we get the next 3x NUM_LEDS bytes and fill up the leds array (in the order r, g, b assuming this was sent that way)

if you don't want to miss the first frame, you can inverse the way you send the frame in Python, send 255 first as a start marker and then the color array.