Simple and Robust {Computer — Arduino} Serial Communication

Projects Networking, Protocols, and Devices
1

Hello,

I would like to present a project I have been working on for the past few years: communicating with the Arduino. Trying to solve my problems, i developped a simple communication protocol and created implementation in different languages (C++, Python, Rust). At the end, this project can also be used to communicate with the Arduino (using serial) but also between two raspberry pi using bluetooth or sockets.

I just finished writing a complete article describing the problem and proposing a solution:

You can read the full article here: Simple and Robust {Computer — Arduino} Serial Communication | by Antonin RAFFIN | Medium

If you have any remark, any question, i would be glad to answer you =).

TL;DR:

Arduino built-in functions for sending/receiving data are not very handy and sturdy. We introduce a protocol to communicate (using serial port, bluetooth or sockets) with the Arduino (but not only) in a simple and robust way. We also release examples in various programming languages so you don’t need to bother about how to implement it

Github repos:

2

Interesting. I have been having a very brief look at your Python version as I wrote a simple Python - Arduino demo myself.

I can't see anything on Github that explains the API for your system.

...R

3

Hello,
The main functions are presented in the main repository.
To sum it up, it is functions to read and write ints of different lengths. The details (arguments required) are almost the same for each language.
For now, there is no html documentation for python but code and examples are documented.

Looking at your code, it seems that you are trying to solve a different problem that is sending strings.

4

araffin:
Looking at your code, it seems that you are trying to solve a different problem that is sending strings.

I find that sending data in human readable form makes debugging very much easier and I always use it unless I need the extra performance of sending binary data - which is very very seldom.

I have had a look at your main link GitHub - araffin/arduino-robust-serial: A simple and robust serial communication protocol. It was designed for Arduino but can be used for other purposes (e.g. bluetooth, sockets). Implementation in C Arduino, C++, Python and Rust. and I find myself more confused than I was before.

I had assumed there would be a .ino file representing the Arduino code and (for example) a .py file representing the Python code.

There is a list of functions on the main link - for example read_i8(). Is that an Arduino function or a Python function, or both?

...R

5

One question, did you read the article ?
It explains both the high level view and go into the details step by step.

Regarding the Arduino code, as written in the readme, it is located in the arduino-board folder. Because i am using Makefile to compile it, the extension is .cpp and not .ino (the file is named slave.cpp)
For the Python implementation, you can look at an example in the python repo.

Robin2:
There is a list of functions on the main link - for example read_i8(). Is that an Arduino function or a Python function, or both ?

Both. That is the all point of having a common protocol, you don't have to care about the language, same functions are provided for each implementation. Once you understand how it works for one language, it is easy to switch to another one.

6

araffin:
One question, did you read the article ?

Yes, I just found it now and had a quick look through it. I don't get it, I'm afraid.

For example I don't agree with this because the example that your logic is based on is designed to support the argument rather than to avoid the problem.

First, there is a lack of readability: looking at the code, you have to understand that the character ‘a’ corresponds to the order of going forward. You have to make sure that at every place, where you want to read or send a command, the same character is used. This becomes clearly unmaintainable and error prone when the number of possible orders grows, or when you want to interface with different programming languages.

And this is a complete non-issue if the Arduino code is designed sensibly

But the main issue is not there. The “classic technique” for arduino serial forgot one important thing: the limited serial buffer size on the Arduino.

Also the whole concept underlying the Arduino system is simplicity. Why would you force people go to the trouble of using a Make file when they could more easily use a .ino file.

...R

7

I have had a bit of spare time so out of curiosity I took the trouble to download some of your Arduino code. I downloaded the 4 files order.h, parameters.h, slave.cpp and slave.h from ..../araffin/arduino-robust-serial

I changed the name of slave.cpp to slave ,ino and it compiled without any error. Then I realized that it does not need the file slave.h so I commented out the line #include "slave.h" and it compiles fine.

I have to say I do not believe your code is as robust as you would like to think because you have no means to identify the start and end of a message or to validate the content of a message. And you attempt to read multiple bytes even though you only check that there is at least one available.

IMHO the system in the 3rd example in Serial Input Basics is considerably more robust as it does not try to interpret any part of a message until it has a complete message. That makes it easy to extend it to include a checksum.

As you mentioned earlier it is designed to work with text rather than binary data but it is equally possible to implement a similar system for binary data. I have also used a system in one of my own projects that converts numbers to a sort of base64 code and sends them as ascii data - 3 characters can represent numbers up to about 266,000. By sticking with human readable characters debugging is much easier and the conversion to "proper" numbers is much faster than the atoi() function.

Another problem I have with your code is the use of wait_for_bytes() to wait for a number of bytes to arrive. Serial is very slow by Arduino standards (even at 500,000 baud) and it is a waste of resources to wait for data to arrive.

...R

8

Robin2:
And this is a complete non-issue if the Arduino code is designed sensibly

Could you develop please ?

Robin2:
Also the whole concept underlying the Arduino system is simplicity. Why would you force people go to the trouble of using a Make file when they could more easily use a .ino file.

Robin2:
I changed the name of slave.cpp to slave ,ino and it compiled without any error. Then I realized that it does not need the file slave.h so I commented out the line #include "slave.h" and it compiles fine.

I use Makefile only for convenience (it removes the Arduino IDE dependency). As you discovered it, you can just renamed it to .ino and it will work.

Robin2:
I have to say I do not believe your code is as robust as you would like to think because you have no means to identify the start and end of a message or to validate the content of a message. And you attempt to read multiple bytes even though you only check that there is at least one available.

Robin2:
Another problem I have with your code is the use of wait_for_bytes() to wait for a number of bytes to arrive. Serial is very slow by Arduino standards (even at 500,000 baud) and it is a waste of resources to wait for data to arrive.

For everything that is start, end and content validation, the low-lewel RS232 serial protocol already implements mechanisms (start bit, end bit and parity bit).
If you keep the same logic between your arduino/computer code (i.e. same number and type for each parameter of an order), you know the start, thanks to the "Order" message and you know the end because of the fixed length of the parameters.
A the very beginning of the communication, there is also an important synchronisation step (the "hello" exchange) that makes it work.

Concerning content validation, as stated in the possible improvements, there is no checksum for now. Also, if there is a shift once, it may shifts everything (because we don't do synchronisation twice). In practice, i never experienced such a problem.

I should have maybe defined what i meant by "robust". I meant "robust" regarding the buffer overflow issue: even when sending lots of messages (e.g. for remotely controlling a car) the communication continues to work and no messages are lost because of that buffer.

wait_for_bytes() is mandatory as you point it out when we want to read multiple bytes. This allow to be sure that we have enough bytes in the buffer before reading a 32-bits int for example.
I don't see that as a waste of ressource because in many cases, you need to receive messages (e.g. motor speed) before continuing the program.

Robin2:
As you mentioned earlier it is designed to work with text rather than binary data but it is equally possible to implement a similar system for binary data. I have also used a system in one of my own projects that converts numbers to a sort of base64 code and sends them as ascii data - 3 characters can represent numbers up to about 266,000. By sticking with human readable characters debugging is much easier and the conversion to "proper" numbers is much faster than the atoi() function.

Well, most of the debugging part was done creating this library: i looked at binary representation of numbers at the beginning and error were mostly easy to spot. Also, numbers (that is the output of the different functions) are still human readable. Finally, this protocol is not an attempt to replace the "Serial.println()" that is really useful for debug.

9

araffin:
Could you develop please ?

Study the code in the 2nd and 3rd examples in Serial Input Basics. By changing the value of the constant numChars any amount of data can be received.

I use Makefile only for convenience (it removes the Arduino IDE dependency). As you discovered it, you can just renamed it to .ino and it will work.

This is the Arduino Forum and your project is aimed at Arduino users so I should not have needed to rename anything.

For everything that is start, end and content validation, the low-lewel RS232 serial protocol already implements mechanisms (start bit, end bit and parity bit).

That is only relevant for a single byte. It does nothing to identify the start or end of a message.

If you keep the same logic between your arduino/computer code (i.e. same number and type for each parameter of an order), you know the start, thanks to the "Order" message and you know the end because of the fixed length of the parameters.
A the very beginning of the communication, there is also an important synchronisation step (the "hello" exchange) that makes it work.

That makes the big assumption that there is never an error in transmission or loss of communication. It is very easy for two systems to get out of sync.

I should have maybe defined what i meant by "robust". I meant "robust" regarding the buffer overflow issue: even when sending lots of messages (e.g. for remotely controlling a car) the communication continues to work and no messages are lost because of that buffer.

I have not seen anything unusual in your program to deal with buffer overflow. As as I can see you just send short messages - which is perfectly sensible, but hardly needs a GIT project.

wait_for_bytes() is mandatory as you point it out when we want to read multiple bytes. This allow to be sure that we have enough bytes in the buffer before reading a 32-bits int for example.
I don't see that as a waste of ressource because in many cases, you need to receive messages (e.g. motor speed) before continuing the program.

I understand that you need to receive all the bytes to make up a multi-byte message. But there is no need to stand around on the street corner chewing gum while that happens. You don't sit in front of the oven while it cooks a chicken.

And, to think about this in terms of controlling a robot - why not use the time between bytes to check for an obstacle, or to check and correct the RPM of the motor.

...R

10

This is the Arduino Forum and your project is aimed at Arduino users so I should not have needed to rename anything.

Following your recommendation, i added files (in fact symlinks) and instructions for Arduino IDE users. :wink:

I have not seen anything unusual in your program to deal with buffer overflow. As as I can see you just send short messages - which is perfectly sensible, but hardly needs a GIT project.

The issue and the proposed solution is described in the section "Solving buffer overflow" of the article. The solution is to acknowledge the reception of each messages, that avoids sending too much messages in a short amount of time. And in my personal experience, that solved the different problems we had (messages lost or Arduino not responding).