Help designing simple serial packet protocol - bytes lost?

This is yet another question about Arduino serial packet protocols. :blush:

I’m looking for something simple suitable to send packets of binary data from one processor (actually the Raspberry Pi) to and from an Arduino (Uno).

All the libraries I’ve found (nanopb, Arduino-serial-packet, Fermata, etc) seem to add a lot of extra value. Fine, but absolutely not what I’m looking for. UComms sounded about right, but has vanished. There’s some good advice at Arduino Serial protocol design patterns – todbot blog , but no implementation.

If I have to write my own, I’m not clear on the requirements. Do bytes ever get lost? I.e. do I have to have some kind of flag byte to resynchronise packet starts (ASP effectively uses carriage return; nanopb seems to assume there’s no lossage)? And if I do that, do I need to escape the flag byte when sending packet data?

  • Charles

CharlesWeir:
Do bytes ever get lost? I.e. do I have to have some kind of flag byte to resynchronise packet starts (ASP effectively uses carriage return; nanopb seems to assume there’s no lossage)? And if I do that, do I need to escape the flag byte when sending packet data?

Yes, data can get lost.

You don't need to escape your synchronization bytes as long as you have some way of detecting a failed transmission. Look for a sync byte only between messages. If the sync byte you find is the start of a new message then you are good to go. If it's in the middle of a failed message you will eventually detect a second failure and go back to looking for sync.

You could implement SLIP (Serial Line Internet Protocol) and TCP on top of that. That will insure your messages get through and remain in order. It also allows multiple separate streams over the same serial connection.

First off, I'd like to mini rant and say that I think this question comes up so often that it seems like there should be a general purpose hobbyist serial protocol by now, like how commercial stuff standardized on USB.

What I have settled on for my stuff is a simple structure where packets have a start byte, an end byte, and an escape that causes the next byte to be interpreted literally and not as an escape. I have another special character that separates the "header" from the "message" and the last 2 bytes of message are the checksum.

I do in fact need to escape my sync bytes because I don't have any other way of detecting the starts or ends of transmissions besides my special flag bytes.

The real trouble is when you need complete reliability and you need to be able to automatically resend if the listener doesn't get the packet. But this doesn't really come up very often in hobby stuff. What I do is include an incrementing in the initial packet that gets echoed back by the listener, the sender will resend if it doesn't get the response, and the listener will ignore duplicates based on the counter.

The other way is the "pull" method where the listener requests a piece of data and repeats the request if needed.

You really do want a checksum. I like the modified Fletcher checksum where you take the mod 256 sum of all the bytes, and you also take the sum of that sum value at every step. Hard to explain but the code is like four lines. I stayed away from CRCs because I wanted the minimum resource usage, but CRCs are more reliable.

Really,there won't be very many errors, so unless every single packet is extremely important(one missed temperature reading an hour out of thousands isn't a big deal to most people), the only thing to worry about is detecting errors(because a bad packet can be worse than a dropped packet) and synchronization errors(because you don't want one bad sequence to cause a locked up state)

That's one of the most important things in protocol design: make sure there is no sequence of data on the wire capable of causing the decoder to enter into an infinite wait period.

A lot of people like to base their packet structure on a length byte near the start of the packet. I'm not much of a fan, partly because IMHO the code to check for an undisclosed ending newline is a lot cleaner looking that "does the total length match the length byte, and have we gotten far enough along in the packet to have actually recieved a length byte. Also, if you do multi master busses with collision detection, start and stop bytes allow instant collision recovery as a start of packet char immediately discards any incomplete data sitting around with no waiting for timeouts.

Some protocols dellimit packets by empty space of a certain length. This would be my personal least favorite because it adds a fairly strict timing dependance.

Thank you.

I agree EternityForest's protocol specification - looks very good. But it does look a bit of work to implement!

I wonder if there's any public domain project that uses a protocol like that, where I could just extract the relevant section of the code?

  • Charles

My protocol library:

There are some drawbacks to using escape bytes, namely that corruption can make a byte look like an escape when it should't, or make an escape into a normal byte, so you really gotta use a checksum(Serial comms are usually reliable so I just use a twos complement fletcher, because it's one of the smallest fastest hashes there is.)