End Of Line Serial Communication

Hi All,

I have run into a very interesting issue. In fact its more my ignorance rather than an issue. So during serial communication between Pi and Arduino, I am using EOL ('\n') character to detect when the data is fully received. On the arduino side, its fine because all of the data are just queries from Pi. However on the Pi side, I am getting sensor readings from arduino. My data format is (byte1, byte2,...): [slaveID, Sensor1 Reading, Sensor2 Reading, EOL]. However, there is a corner case when the sensor data contains a byte that is similar to that of EOL i.e. 0x0A. In that case, the serial.readline() (python language) command stops reading the moment it sees 0x0A. How to people tackle this issue ?

Arduino Nano, Pi Model 2, I am using level shifter and all the other protection mechanisms.

Some possible solutions:

Send the data as text using numeric characters. This is pretty inefficient, but it has some great advantages, such as human-readability, and the fact that you don't have to worry about Endianness. On the flip side, you have to send much more data, and printing/parsing the data takes more time than just reinterpreting the raw bytes.
See Robin2's Serial Input Basics.

Convert the data to a known range of characters using base64, for example. Encoding and decoding still takes some time, and Endianness can be a problem. But you can fit more information into every byte you transmit.

Use escape bytes for your end markers. A very clear example is the SLIP protocol: RFC1055
Instead of sending END as a data byte, you send two bytes: ESC ESC_END. Now you have to escape ESC as well: send ESC ESC_ESC instead of ESC as a data byte.

Pieter

mqureshi9:
Hi All,

I have run into a very interesting issue. In fact its more my ignorance rather than an issue. So during serial communication between Pi and Arduino, I am using EOL ('\n') character to detect when the data is fully received. On the arduino side, its fine because all of the data are just queries from Pi. However on the Pi side, I am getting sensor readings from arduino. My data format is (byte1, byte2,...): [slaveID, Sensor1 Reading, Sensor2 Reading, EOL]. However, there is a corner case when the sensor data contains a byte that is similar to that of EOL i.e. 0x0A. In that case, the serial.readline() (python language) command stops reading the moment it sees 0x0A. How to people tackle this issue ?

Arduino Nano, Pi Model 2, I am using level shifter and all the other protection mechanisms.

No, not your ignorance at all. Even the big corporations making communications equipment have not discovered this type of error until the equipment was released to the field.

Paul

Since you have a binary format already, does it have a fixed number of bytes? Then you can just count bytes until you get to where you expect to find the \n. If you don't find it, try to re-sync on any \n that you previously thought was part of the packet. If there's no \n at all then the packet is unrecoverable and you wait for the next packet.

JSON reply and a text command are how I am doing it. So when I ask for the ID it looks like

/0/id?/n
{"id":{"name":"Reflow","desc":"RPUno (14140^9) Board /w atmega328p","avr-gcc":"5.4.0"}}/n

I hold the command in a command buffer and spin loop until the chunks of JSON are done outputting. Each chunk is less than 32 bytes since that is the AVR UART buffer size. I spin loop waiting for the buffer to be empty before filling it with the next JSON chunk (a full buffer will block execution).

The "/0" both have to be sent before an echo will begin, which has to do with my wacky setup which I will ignore hear.

JSON goes into Python fairly well, this Python program shows the interaction.

First, catch the command line interface by forcing it into a known state. Then send commands to load data into EEPROM for a reflow profile.

Since you have a Raspberry Pi you could probably build and bootload the Reflow firmware onto an Uno to try out the interaction (it uses Optiboot like Arduino IDE, but without the IDE). The Makefile assumes the Pi bootloads over its hardware UART with RTS/CTS handshaking enabled (RTS is used in place of DTR to do the reset trick). I have some notes for enabling RTS/CTS which also needs a level shift.

You can also use picocom to interact with the Uno from the Pi's hardware UART.

picocom -b 38400 /dev/ttyAMA0
...
Terminal ready
/0/id?
{"id":{"name":"Reflow","desc":"RPUno (14140^9) Board /w atmega328p","avr-gcc":"5.4.0"}}

This Python - Arduino demo may be useful.

I much refer to send data in human readable form unless there is a performance requirement that precludes it. It makes debugging much easier.

...R