Pages: [1] 2   Go Down
Author Topic: Serial protocol how-to  (Read 2450 times)
0 Members and 1 Guest are viewing this topic.
Grenoble/Lyon - France
Offline Offline
Sr. Member
****
Karma: 0
Posts: 363
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Hey there

I have a processing sketch that loads a black/white 8*8px animated gif, analyze each frames and sends the value corresponding to the lit pixels (8 bytes, one per line) to the arduino board, approximately 25 times per second (maybe more). On the arduino side, I have a max7221 hooked up to a 8*8 matrix.

After a few sends, my data gets f****ed up, certainly because I don't have a proper serial protocol. I can't use libraries like firmata or SMS, because ultimately, this prototype sketch I'm doing for a friend will be recoded on a PIC mcu.

How would you implement a communication protocol that can run for approximately 2 hours at (let's say) 50 frames per second (3200bauds of pure data, without the header and stuff for the protocol) without any bug ? ^^

Thanks for any infos

melka
Logged


Grenoble/Lyon - France
Offline Offline
Sr. Member
****
Karma: 0
Posts: 363
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

You know what ? I think I got it. On the arduino side, I just make sure that I have 8 byte in the buffer before reading it, then I flush. I clear the Processing buffer after 8 sends too. Seems to work.

I'm open to any other input, though
Logged


New Jersey
Offline Offline
Full Member
***
Karma: 0
Posts: 193
Ard at work
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

This may be a stupid suggestion, but I've had good results taking a leaf from GPS NMEA strings. I've just never tried it quite that fast (more like 9 bytes 5 times a second).

I start a sentence with a special character, say '$', then two-three characters describing the sentence structure if there are several possibilities, then a series of values separated by commas, and stop a sentence with a carriage return. My values uses three bytes and only one sentence structure, but in your case a comma every two or four bytes makes more sense, with perhaps a different sentence header for each line of the matrix?

You can then use the GPS NMEA string reading code from the examples with some modifications. You get the occasional garble, but it resets at every sentence. Your code could stick incoming serial data into an array until it gets a carriage return, recognize the header to ID the line, then split it at the commas, and finally decode the values between the commas. Update matrix, start over with contents of serial buffer.

I dunno if this would work fast enough for you, tho.
Logged

----------
Mathieu

Grenoble/Lyon - France
Offline Offline
Sr. Member
****
Karma: 0
Posts: 363
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

One thing that I don't really undertand is that I send 8 bits integers on my line, so they varry from 0 to 255, obviously. Won't the $ character or the carriage return be interpreted as their ASCII equivalent, hence 13 for the carriage return if i'm correct. But 13 will light me leds 1, 3 and 4, no ?

I'll give it a try, right now I managed to do what I wanted (with some weird results from the proMIDI library of processing, but well, at least it's not the Arduino ^^) and it works as a prototype, so for the time being, I'll leave it as is.
But for the final version (something like two 24*24 white leds matrix to control), I'll give it a go, for sure.
Logged


0
Offline Offline
Faraday Member
**
Karma: 8
Posts: 2526
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
the carriage return be interpreted as their ASCII equivalent, hence 13 for the carriage return if i'm correct.
That one can be tricky, depending on the OS, terminal application, or device you are talking to. Some use a linefeed (ASCII 10), some use CR, some use a CRLF pair.

But in general, yes you are correct.  The data sent down the wire is the binary number representing the ASCII code for that character - that's pretty much what ASCII is all about.

-j

Logged

New Jersey
Offline Offline
Full Member
***
Karma: 0
Posts: 193
Ard at work
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Depending on how much overhead you have left in your bandwidth, you might want to NOT send your data as bytes but as chars (Three bytes per byte, so not super efficient) or to reserve certain values for control characters (might be difficult for you since you have exactly 8 bits of data per line) or to preface each pair of value bytes with, say, a byte set to zero and interpret each triplet as a three-byte number and anything over 65,535 is garbled data.

You lose bandwidth but you gain legibility both for a human peeking at the exchange and for the arduino.
Logged

----------
Mathieu

Grenoble/Lyon - France
Offline Offline
Sr. Member
****
Karma: 0
Posts: 363
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Maybe I should use the same structure as the monome 40h or series protocol.
http://docs.monome.org/doku.php?id=tech:protocol:40h
http://docs.monome.org/doku.php?id=tech:protocol:serial:series

I already implemented the series serial protocol on another project, and I didn't have any problem at 115200bps.
It takes 2 bytes per row (16 bytes for the whole screen) and works well.
Logged


New Jersey
Offline Offline
Full Member
***
Karma: 0
Posts: 193
Ard at work
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Hey, thanks for sharing those, I didn't know them! :-)
Logged

----------
Mathieu

0
Offline Offline
Full Member
***
Karma: 0
Posts: 207
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The way to send binary data over a serial link is called framing. One uses an "escape" character, DLE (value 0x10) by convention, to "escape" the control elements of the framing.

Usually a frame would take the form of a start byte (STX) and an end byte (ETX) with values 0x02 and 0x03 respectively.

As an example, the following data:

A DLE B ETX DLE STX E

would be framed and sent as:

DLE STX A DLE DLE B ETX DLE DLE STX E DLE ETX

Google DLE STX brings up some relevant documents.

Mike

Logged

Manchester (England England)
Offline Offline
Brattain Member
*****
Karma: 626
Posts: 34128
Solder is electric glue
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

So how do you make sure that the STX and ETX are not incidentally in the data you send?

What you need is a unique code which is difficult for 8 bits of data.

One trick you might like to use is to transmit a synchronising start byte as some value say 0x02 BUT transmit it using the wrong parity, then when you get a 0x02 with the wrong parity you know you have a valid start and you can read the next n bytes knowing their sequence.

Failing that your best spiting it up into nibbles and using the top bits to uniquely identify data and header.
Logged

0
Offline Offline
Faraday Member
**
Karma: 8
Posts: 2526
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
So how do you make sure that the STX and ETX are not incidentally in the data you send?
By quoting with the DLE.  The real problem is what happens when DLE is in your data; that solution is to "byte stuff" - replace a single DLE with two consecutive DLEs on TX.

-j

Logged

0
Offline Offline
Full Member
***
Karma: 0
Posts: 207
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Mike,

kg4 has it right.  This DLE STX thing is suitable for sending binary data - all 256 combinations - is commonly used and is robust.

I have implemented it several times in the past and included checksums to test for data integrity, and in one application I included code to facilitate data correction.

Regards,

Mike
Logged

Austin, TX USA
Offline Offline
God Member
*****
Karma: 4
Posts: 997
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

First of all, it's good to see both Big and Grumpy in the same thread. smiley

I've been thinking about how the Arduino HardwareSerial class is pretty good for formatted text, but it would be nice to extend it for binary data.  Wouldn't it be useful to include, in addition to the print* suite of functions, something like:

Code:
void HardwareSerial::writebuf(byte *buffer, int bufsize);
void HardwareSerial::readbuf(byte *buffer, int *bufsize);

// sample usage:
byte buf[64];
Serial.writebuf(buf, sizeof(buf));

int size = sizeof(buf);
Serial.readbuf(buf, &size);

For the readbuf method, you would send the maximum size in and then the actual size would be returned.  Doesn't this seem like it would useful?  I would be happy to contribute to the implementation, along the lines of the discussion above.

Mikal
« Last Edit: September 15, 2008, 12:22:06 pm by mikalhart » Logged

London
Offline Offline
Tesla Member
***
Karma: 10
Posts: 6255
Have fun!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
First of all, it's good to see both Big and Grumpy in the same thread.

I've been thinking about how the Arduino HardwareSerial class is pretty good for formatted text, but it would be nice to extend it for binary data.  ... Doesn't this seem like it would useful?  I would be happy to contribute to the implementation, along the lines of the discussion above.

Mikal

It's good to see all the above posters here together in one thread  smiley-wink

Fwiw, I haven't seen a big need for methods to send and receive raw binary  packets on the serial port. What I have noticed is a need for an easy way to handle messages that  include parameters of various types. At one point I did write a protocol handler that passed bytes, ints, arrays  and string parameters over the hardware serial port and/or i2c. I may have included the code in a thread a few months ago but felt it needed a lot of work to make it easy enough for general use.

But I would be happy to work with you guys on something along the lines of a simple and extendable protocol handler for arduino/arduino and arduino/pc messaging if there was consensus  that this is something the Arduino community needs and someone could outline the requirements.
Logged

Grenoble/Lyon - France
Offline Offline
Sr. Member
****
Karma: 0
Posts: 363
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Nice to see that this thread took this direction ^^
Logged


Pages: [1] 2   Go Up
Jump to: