Serial protocol how-to

Hey there

I have a processing sketch that loads a black/white 88px animated gif, analyze each frames and sends the value corresponding to the lit pixels (8 bytes, one per line) to the arduino board, approximately 25 times per second (maybe more). On the arduino side, I have a max7221 hooked up to a 88 matrix.

After a few sends, my data gets f****ed up, certainly because I don’t have a proper serial protocol. I can’t use libraries like firmata or SMS, because ultimately, this prototype sketch I’m doing for a friend will be recoded on a PIC mcu.

How would you implement a communication protocol that can run for approximately 2 hours at (let’s say) 50 frames per second (3200bauds of pure data, without the header and stuff for the protocol) without any bug ? ^^

Thanks for any infos

melka

You know what ? I think I got it. On the arduino side, I just make sure that I have 8 byte in the buffer before reading it, then I flush. I clear the Processing buffer after 8 sends too. Seems to work.

I’m open to any other input, though

This may be a stupid suggestion, but I’ve had good results taking a leaf from GPS NMEA strings. I’ve just never tried it quite that fast (more like 9 bytes 5 times a second).

I start a sentence with a special character, say ‘$’, then two-three characters describing the sentence structure if there are several possibilities, then a series of values separated by commas, and stop a sentence with a carriage return. My values uses three bytes and only one sentence structure, but in your case a comma every two or four bytes makes more sense, with perhaps a different sentence header for each line of the matrix?

You can then use the GPS NMEA string reading code from the examples with some modifications. You get the occasional garble, but it resets at every sentence. Your code could stick incoming serial data into an array until it gets a carriage return, recognize the header to ID the line, then split it at the commas, and finally decode the values between the commas. Update matrix, start over with contents of serial buffer.

I dunno if this would work fast enough for you, tho.

One thing that I don’t really undertand is that I send 8 bits integers on my line, so they varry from 0 to 255, obviously. Won’t the $ character or the carriage return be interpreted as their ASCII equivalent, hence 13 for the carriage return if i’m correct. But 13 will light me leds 1, 3 and 4, no ?

I’ll give it a try, right now I managed to do what I wanted (with some weird results from the proMIDI library of processing, but well, at least it’s not the Arduino ^^) and it works as a prototype, so for the time being, I’ll leave it as is.
But for the final version (something like two 24*24 white leds matrix to control), I’ll give it a go, for sure.

the carriage return be interpreted as their ASCII equivalent, hence 13 for the carriage return if i’m correct.

That one can be tricky, depending on the OS, terminal application, or device you are talking to. Some use a linefeed (ASCII 10), some use CR, some use a CRLF pair.

But in general, yes you are correct. The data sent down the wire is the binary number representing the ASCII code for that character - that’s pretty much what ASCII is all about.

-j

Depending on how much overhead you have left in your bandwidth, you might want to NOT send your data as bytes but as chars (Three bytes per byte, so not super efficient) or to reserve certain values for control characters (might be difficult for you since you have exactly 8 bits of data per line) or to preface each pair of value bytes with, say, a byte set to zero and interpret each triplet as a three-byte number and anything over 65,535 is garbled data.

You lose bandwidth but you gain legibility both for a human peeking at the exchange and for the arduino.

Maybe I should use the same structure as the monome 40h or series protocol.
http://docs.monome.org/doku.php?id=tech:protocol:40h
http://docs.monome.org/doku.php?id=tech:protocol:serial:series

I already implemented the series serial protocol on another project, and I didn’t have any problem at 115200bps.
It takes 2 bytes per row (16 bytes for the whole screen) and works well.

Hey, thanks for sharing those, I didn’t know them! :slight_smile:

The way to send binary data over a serial link is called framing. One uses an “escape” character, DLE (value 0x10) by convention, to “escape” the control elements of the framing.

Usually a frame would take the form of a start byte (STX) and an end byte (ETX) with values 0x02 and 0x03 respectively.

As an example, the following data:

A DLE B ETX DLE STX E

would be framed and sent as:

DLE STX A DLE DLE B ETX DLE DLE STX E DLE ETX

Google DLE STX brings up some relevant documents.

Mike

So how do you make sure that the STX and ETX are not incidentally in the data you send?

What you need is a unique code which is difficult for 8 bits of data.

One trick you might like to use is to transmit a synchronising start byte as some value say 0x02 BUT transmit it using the wrong parity, then when you get a 0x02 with the wrong parity you know you have a valid start and you can read the next n bytes knowing their sequence.

Failing that your best spiting it up into nibbles and using the top bits to uniquely identify data and header.

So how do you make sure that the STX and ETX are not incidentally in the data you send?

By quoting with the DLE. The real problem is what happens when DLE is in your data; that solution is to “byte stuff” - replace a single DLE with two consecutive DLEs on TX.

-j

Mike,

kg4 has it right. This DLE STX thing is suitable for sending binary data - all 256 combinations - is commonly used and is robust.

I have implemented it several times in the past and included checksums to test for data integrity, and in one application I included code to facilitate data correction.

Regards,

Mike

First of all, it’s good to see both Big and Grumpy in the same thread. :slight_smile:

I’ve been thinking about how the Arduino HardwareSerial class is pretty good for formatted text, but it would be nice to extend it for binary data. Wouldn’t it be useful to include, in addition to the print* suite of functions, something like:

void HardwareSerial::writebuf(byte *buffer, int bufsize);
void HardwareSerial::readbuf(byte *buffer, int *bufsize);

// sample usage:
byte buf[64];
Serial.writebuf(buf, sizeof(buf));

int size = sizeof(buf);
Serial.readbuf(buf, &size);

For the readbuf method, you would send the maximum size in and then the actual size would be returned. Doesn’t this seem like it would useful? I would be happy to contribute to the implementation, along the lines of the discussion above.

Mikal

First of all, it’s good to see both Big and Grumpy in the same thread.

I’ve been thinking about how the Arduino HardwareSerial class is pretty good for formatted text, but it would be nice to extend it for binary data. … Doesn’t this seem like it would useful? I would be happy to contribute to the implementation, along the lines of the discussion above.

Mikal

It’s good to see all the above posters here together in one thread :wink:

Fwiw, I haven’t seen a big need for methods to send and receive raw binary packets on the serial port. What I have noticed is a need for an easy way to handle messages that include parameters of various types. At one point I did write a protocol handler that passed bytes, ints, arrays and string parameters over the hardware serial port and/or i2c. I may have included the code in a thread a few months ago but felt it needed a lot of work to make it easy enough for general use.

But I would be happy to work with you guys on something along the lines of a simple and extendable protocol handler for arduino/arduino and arduino/pc messaging if there was consensus that this is something the Arduino community needs and someone could outline the requirements.

Nice to see that this thread took this direction ^^

The real problem is what happens when DLE is in your data; that solution is to “byte stuff” - replace a single DLE with two consecutive DLEs on TX.

So what happens when there is two consecrative DLEs in the data. Sure it won’t happen often but it will happen some time. One of my favourite miss quotes from Terry Pratchit is that “Engineers know million to one chances happen nine times out of ten”

The only real way to have something robust is to make sure it is something that can never occur in random binary data. Of course that involves “wasting” bandwidth and the compromise points change as the packet size (useful data) increases. So there would be no best one size fits all solution anyway. Then there is the added pay load that error detection / correction could bring.

It would be good to have a common protocol for stuff like this. I am up for cooperating on this one. :slight_smile:

So what happens when there is two consecrative DLEs in the data.

Then the data stream will see 4 DLEs, and the receive end will toss the two extra “quote” DLEs.

This is an old and reasonably well known protocol, as evidenced by the fact that specific ASCII control characters exist to support it. BigMike didn’t just make it up (or maybe he did and we’re truly fortunate to have such a venerable master among us. :slight_smile: )

-j

OK sorry now I get it. It might be old (and so am I ) but I have not come across that before.

For those not in the know then this RFC 264 - The Data Transfer Protocol (RFC264) should make things straight.

Forgive me if I reiterate the essence of this just to see if I have this right. When sending data you never send a single DLE it is always doubled up. When not sending data waiting for synchronisation you can send a single DLE.

At the receive end if you see a double DLE you know you are in the data portion and can wait for it to finish. If you see a single one you know you are in the header.

The only problem is that you can’t process the data as it comes in if it is a DLE but have to wait until the next byte before you know if it is a piece of data that looks like a DLE or the end of the sequence.

This points to a shortcoming on this board namely the lack of an smilie with a rising hat.

Thanks for putting me right.

Mike,

As kg says, this is a standard mechanism for transferring binary data, not just something I dreamt up. It is robust and will handle all manner of random binary data that you throw at it.

That said, it may not be optimum for all applications - for example if there are a lot of DLEs in the data it may cause the data stream to be larger than is desirable.

“Engineers know million to one chances happen nine times out of ten”

LOL

Regards,

Mike

Edit - Sorry, didn’t see your next post before I posted this.