Arduino serial messaging protocol

Scratching your own itch...
As a university lecturer, I was annoyed by the fact that, every year, all my students were implementing their own very simple protocols to connect Arduino & pc applications.
As a results it was hard to reuse their code and most of the time it is hard to interconnect applications from different teams.

Why this can be useful
This library was developed to allow multiple teams of students to work together on Arduino/embedded Linux projects
Some teams working on the embedded Arduino side, other teams working on Python applications running on the Raspberry PI.
The goal is to have a standardised "communication protocol" to ensure interoperability between software/hardware blocks.
Hopefully this will allow future student to start from the current code base to develop their projects.

The concept
We are developing a library to exchange short messages (sensordata, commands) between an Arduino and software applications running on a PC. (Linux, embedded Linux, Windows, OS X)

Both sides are able send and receive a range of "standardised" messages over the serial port. All communication is done by sending short human readable ASCII messages. We define several standard command and data packet IDs to provide interoperability between different applications.

The protocol was designed to be relatively easy to comprehend and process by both human and computer.

• A standardised message structure (more info below)
• All fields in a message are separated by delimiters (just a letter indicating what the next field will be)
• All communication is in HEX ASCII values (to allow human and computer to understand the packets)
• Invalid messages are detected by calculating the parity (XOR all serial bytes)

The Message Types
The library support seven basic packet types.

• "Command": start motor, blink LED, go to sleep mode,...
• "Command Reply": e.g. motor started, LED blinking, going to sleep mode now,...
• "8-bit Data": (standard Arduino Byte type): e.g. temperature is 25 C, distance is 50 cm, humidity is 56%
• "16-bit Data": (standard Arduino int type): e.g. temperature is -25 C, distance is 310 cm, time is 16200 seconds, ...
• "Data array": send multiple sensor measurements in one burst (useful when sampling very rapidly)
• "Data request": e.g. send me the temperature, distance, humidity, ...
• "Data array request": e.g. send me a burst of measurements

Github page: GitHub - jeroendoggen/Arduino-serial-messaging: Library to exchange short messages (sensordata, commands) between an Arduino and a software application running on a PC. (Linux, embedded Linux, Windows, OS X)

Have you seen Firmata?

You might find it of interest for addressing this problem:
http://firmata.org/wiki/Main_Page

@doggenj, sounds like an excellent idea to have your students all using a common protocol, so everyone is on the same wavelength, so to speak [also, of course, the instructor's job is 1000% easier, :-)]. I'm not familiar with Firmata, so cannot comment on that, but am in the process of developing command protocols for my robot systems.

With respect to your protocol, I have 2 initial comments from viewing your overview page,
http://jeroendoggen.github.io/Arduino-serial-messaging/

1. you mention "All communication is done by sending short human readable ASCII messages", but your commands say "00h up to 7Fh", and

ID (hex) Usage
00 Request the version of the protocol

etc. It seems to me these are raw data values, and not ASCII, so not humanly readable. Unless what you're actually sending is '0','0', etc. Can you explain?

2. your page also mentions sending "data packets", but there is no obvious description of what a data packet looks like. . It would be nice to see the exact format of a complete message packet in the overview, and without having to reverse-engineer the source code from github.

Thanks.

I have checked Firmata, but as far as I understand its main goal is to replicate the low-level Arduino API on the PC side.
From the Firmata site: The aim is to allow people to completely control the Arduino from software on the host computer
For example:

• send the result of "AnalogRead" over the serial port when the PC requests this value.
• set the PWM value for the analog output pins

Since this would really lower the maximum sampling rate, this approach does not fit all possible applications.

Our goal is to implement "high level messages" to share information that might be based on the data of multiple sensors. (or multiple devices in a wireless sensor network)
For example:

• This is the current temperature
• Room x is currently not occupied
• Remote controlled car has entered room x

@oric_dan: thanks for the remarks, I will add some extra info to the Github page.

About: "the human readable ASCII messages"

We actually send the ASCII values of hex characters between 0 and F (zero and F) (that implies being very inefficient, 240 unused values)
For example:
The packet: "T01N00I12PffQ21"

Would be 15 bytes on the wire (ASCII : HEX)
T: 54
0: 30
1: 31
N: 4E
0: 30
...

The downside:

• efficiency.
• it is quite simplistic

The upside is:

• Easy to parse (easy to see where a message starts and ends)
• Easy to read (mainly because of the delimiters)
• No problems with magic number or magic sequences for message headers

Thanks for your post, and for sharing your project. It's an interesting problem, isn't it?

The approaches taken by Arduino-serial-messaging and Firmata are both valid and effective, but may I suggest you also consider Bitlash? (http://bitlash.net) Bitlash is an interpreted language that runs on the Arduino and provides the affordances of a mini development environment. Having a development capability on the device offers arbitrary extensibility at run time, and the control "protocol" is just statements in the Bitlash language, sent from the PC. This offers excellent visibility into the communication flow, without special tooling, which is arguably a strong plus in an educational environment.

Bitlash is syntactically a stepping-stone to C, is easy to teach, and easy to learn. If distributed system design is part of your curriculum, it puts enough remotely-programmable intelligence on the device that you can explore a spectrum of options as to partitioning functionality between the PC and the endpoints.

Documentation at http://bitlash.net, code at GitHub - billroy/bitlash: Bitlash: a programmable command shell for arduino

Kind regards,

-br

billroy, thanks for the mention of bitlash, may be of use.

doggenj, thanks for the further explanation, although I don't quite see where the delimiters are:

"T01N00I12PffQ21" .... Easy to read (mainly because of the delimiters)

I've just been working on some code for an RS232 network for my robots, where several nodes can all share the same Rx,Tx pins. One board is master, and several others are slaves, with their Tx pins wire-ANDed together to the master's Rx pin. Then, I'm using a s.w. node-addressing scheme, ala the following. All data in ASCII format, so can enter via the Serial Terminal for testing.

Command Format: 
  1. general msg:  "*a:<cmd>;<cmd><CR>", where '*' = header_char, 'a' = node_address(1,2,3...),
     ';' delimits the commands, and <CR>=carriage return.
  2. <cmd>:  "c<arg1>,<arg2>,...", where 'c'=command_character,
     and <args> = signed-integers.
  3. example:  "*4:a2;s1,-45;g;m64,128<CR>" means attention node=4, read analog 2, set servo 1 = -45 degrees, 
     read GP2D12 sensor, and set motor speeds(left,right) = 64 and 128.

Once an entire msg, ending in , is received into a buffer, I parse it and execute each command in order. I decided I needed to make the inter-command vs intra-command delimiters more obvious, to make the parsing easier.

The delimiters are the non-hex characters: T, N, I, P, Q

Command message T01N00I12PFFQ21 : "set motor speed of 'Arduino zero' to +100%:

• T01: Type 01: Command message
• N00: Number 00: Node number 00 (is the destination)
• I12: CommandID 12: Set motor speed
• PFF: Payload FF: full speed (range: 0 (reverse) -> 80 (stopped) -> FF (forward))
• Q21: Quality 21: parity byte is 21

Data message T12N00I10P08Q0A : "temperature of 'Arduino zero' is 8 degrees"

• T12: Type 12: Data message (1 byte payload)
• N00: Number 00: Node number 00 (is the source)
• I10: SensorID 10: Temperature
• P08: Payload 08: 8 degrees
• Q0A: Quality 0A: parity byte is 0A

doggenj:
I have checked Firmata, but as far as I understand its main goal is to replicate the low-level Arduino API on the PC side.

you might want to check out configurable-firmata that goes beyond that initial goal by providing an api to plug-in feature-classes that implement sensor- or actor specific messages: arduino/examples/ConfigurableFirmata at configurable · firmata/arduino · GitHub

With my scheme, I put the node value at the beginning, eg "4", to limit the amount of extraneous filtering the slave nodes need to do. This way, the non-addressed nodes need only parse to the "4" and can then ignore received chars until another header char, "", is received.

Also, thanks for the links to bitlash and formata. A lot of work there. I decided it's quicker to write my own parser and go on, than to go through the learning curve on them, :-).

doggenj:
I have checked Firmata, but as far as I understand its main goal is to replicate the low-level Arduino API on the PC side.
From the Firmata site: The aim is to allow people to completely control the Arduino from software on the host computer
For example:

• send the result of "AnalogRead" over the serial port when the PC requests this value.
• set the PWM value for the analog output pins

Since this would really lower the maximum sampling rate, this approach does not fit all possible applications.

Our goal is to implement "high level messages" to share information that might be based on the data of multiple sensors. (or multiple devices in a wireless sensor network)
For example:

• This is the current temperature
• Room x is currently not occupied
• Remote controlled car has entered room x

When it comes to interoperability I would always consider using formalized, widely accepted standards first. Being a professional software developer/architect I have to deal with a host of libraries, frameworks, standards etc. When they do not suit my needs, I look for an existing technology I can extend or adapt.

On top of this I believe it is good to teach students common practices and techniques and learn them to adhere to them, at the same time avoiding a NIH-mentality and reinvention of wheels with no need.

In Formata there are a few messages implementing the I2C protocol, a well established industry standard. The I2C request and reply methods allow for sending an arbitrary number of bytes, so hexadecimal tokens would perfectly fit in there as well. However, when programming microcontrollers I personally would prefer keeping messages binary, as it is the nature of hardware programming.