Distinguish between a byte that indicates the beginning of a serial packet

Using Arduino Programming Questions
1

How do you distinguish between a byte that indicates the beginning of a packet and the actual data? I prepend "<" and append ">" to my packets, and the data inside is 10 x 3 x 1 bytes of color information. My problem seems to be that some of this color information equals either "<" or ">", causing packets to be lost. I thought I can get around this in the code below by changing the 4th line to if(!started && inChar == "<") but then if the end of packet character is never received for some reason we are stuck.

How should I approach this?

 while(Serial.available() > 0)
  {
    byte inChar = Serial.read();
    if(inChar == "<")
    {
       index = 0;
       inData[index] = '\0';
       started = true;
       ended = false;
    }
    else if(inChar == ">")
    {
       ended = true;
       break;
    }
    else
    {
      if(started && index < 79)
      {
        inData[index] = inChar<<2;
        index++;
        inData[index] = '\0';
      }
    }
  }
2

Start and end of packet markers must be unique from the data being sent.

Unless there is some terrific need for speed, you will need to send the data in ASCII form, so that the start and end markers ARE unique.

If there is a need for speed, you will need to send some unique value between the bytes that you use a sync byte. For instance, 0, 34, 78, 126, 0, ... Look for the 0, read and store 5 bytes. If the 5th is a zero, too, you can have some (small level of) confidence that the 3 middle bytes represent good data. Use them, and delete the last 4 bytes of the array. Read the next 4. If the 5th is a zero, too, you can have some (small level of) confidence that the 3 middle bytes represent good data.

This is not as good as sending ASCII data, with unique start and end markers, but, it is faster. It all depends on whether speed or accuracy is your top concern.

3 
 byte inChar = Serial.read();
    if(inChar == "<")

A single character is never going to equal a pointer to a constant string.

Single quotes for single characters.

4

Start and end of packet markers must be unique from the data being sent.

I guess for simplicity I could always clamp my color values to say 253, then my start and end of packet markers could be 254 & 253. Would that work or cause issues?

Using the sync byte method is possibly making this project hard for anyone else to understand, but I like the idea and will experiment elsewhere.

5

A single character is never going to equal a pointer to a constant string.

sorry that was meant to be

if(inChar == SOP)

where:

#define SOP '<'
6

If your data is truly binary there is no simple (and reliable) way to get into sync with packets, special break bytes or timing to detect an inter-frame period is the only way I can think of.

I'd go the ASCII route and make life easy on yourself :slight_smile:

That said I can't see any problems with your idea of using two special values.

One advantage to the ASCII option is that it provides a further level of error detection because there are only 16 valid values for bytes. If you go binary you probably should also look at a checksum or CRC.

Rob

7

That said I can't see any problems with your idea of using two special values.

It seems to work quite nicely by clamping my RGB values to 253.
Then my packet markers using:
#define SOP 254
#define EOP 255

It also fixed an annoying flicker bug - I guess before a 255 color value was too much...

I would usually go the ASCII route but I didnt think all my data would fit into one packet, and thought this would be simpler...

8

If you didn't want to sacrifice values, you could designate one value as an "escape" byte, which signifies that the following character should be treated as a special case.

E.g., if you had 255 as your escape value:

">" = data (62)
255 ">" = end packet
255 255 = data (255)

etc.

9

Maybe this will provide some inspiration...

10

If you can make up your own format and speed is not an issue why not just send your data as hex value of the byte using ascii printing characters. So you don't send 1C as a single binary byte you send the two printable characters '1' 'C'. You then end each transmission with carriage return line feed.
What you will be transmitting will then be fixed length lines of characters that you can read by eye.
You could also add a space between the values just to make it even easier to manually read.
This makes for simple debugging and also avoids potential problems with "special" characters.
Synchronising on the CR LF should be easy.

11

I know this is lazy, but it works for me.

Research easy transfer from bill porter. Either on serial or I2C. Works great, abstracts all the stuff, includes CRC check, etc. I love it.

12

Here it is:

Looks nice, I'll have to test it some day soon! :slight_smile:

13

tuxduino:
Looks nice, I'll have to test it some day soon! :slight_smile:

IMO, this is a totally under-appreciated work, successfully used it to communicate between two arduinos on a PCB @ 1Mbit/s. Also use it to communicate between arduinos on a RS485 bus. The only 'limitation' is that you have define up front what is going to be sent. Thus, this is not as flexible as some would perhaps like (i.e. more flexibility re: changing the data types, quantity being sent from inside the loop() program rather than having to define the structure up front). But for a well-defined set of data being exchanged, this neat software is the answer to folk like me who want to focus more on manipulating the data rather than how to transmit it reliably.

Naturally, there are many good reasons to come up with your own system, should you have the expertise to pull it off. I don't have the time or the background to be really good at CRC checks, flexible data strings, etc. Then there are those, for whom coming up with their own flavor of EasyTransfer is part of the journey. My hat goes off to them!

But, should you find EasyTransfer to be the answer to some of your prayers, consider donating something to Bill, he's about to get married!

14

But, should you find EasyTransfer to be the answer to some of your prayers, consider donating something to Bill, he's about to get married!

Yes, it's been my experience that marriage does require a sufficient increase in available funds. :wink:

I also agree that EasyTransfer library is a great contribution to the Arduino community and warrants wider exposure.

Lefty

15

Thanks Constantin, your words definitely increased the probability that I will actually try that library :stuck_out_tongue:

16

IMO, this is a totally under-appreciated work,

I also agree that EasyTransfer library is a great contribution to the Arduino community and warrants wider exposure.

I don't think I've ever had a forum post make me blush before. Thanks guys. I should really check these forums more often. Keeping up with the support emails/comments on my own site is almost a part time job as it is. I've been getting 15k hits a month just on my 3 main Arduino libraries.

It's funny that Easy Transfer started as just some idea I had on a Sunday and threw it together in a few hours. I was actually initially embarrassed when the main blogs wrote about it. I didn't feel it did anything amazing or really worthwhile. Reading through the Arduino forum now I see it's really getting some use.

FYI, wedding is in 2 months. I'm ready to skip it for the courthouse. Worst part is dealing with the parental units. Wish me luck.

-Bill

17

This is but the beginning. You may remember a comment in the past about weddings being perfect time sinks, i.e. whatever available time you have will be sucked up, and spent. Best of luck with it all, and thanks again for making such an awesome piece of code that does what it says and perfectly.

FWIW, I am switching to the Teensy 3 (i.e ARM) platform for some of my boards, I presume that Easytransfer should compile just fine on the Teensy 3?

And if it doesn't, can I send you one? These are extremely nifty little platforms.

PS: Donated to your beer errr... wedding fund. That's right, it was the wedding fund.

18

Haha! man you weren't kidding.

I have a teensy 3 from the initial kick starter campaign. I just won't be able to touch it until after the wedding. The only part that might not make it compile is if the Teensy port changed the HardwareSerial object type at all, which I doubt. It's the only part of the library that really interfaces with anything else in the environment.

And thanks for the gift, you are too generous.

If the beer helps me survive the wedding, it should come out of the wedding fund, right? :smiley:

19

The only part that might not make it compile is if the Teensy port changed the HardwareSerial object type at all, which I doubt.

Not in any way that affects Arduino(like) code that access the serial port.