Help designing simple serial packet protocol - bytes lost?

First off, I'd like to mini rant and say that I think this question comes up so often that it seems like there should be a general purpose hobbyist serial protocol by now, like how commercial stuff standardized on USB.

What I have settled on for my stuff is a simple structure where packets have a start byte, an end byte, and an escape that causes the next byte to be interpreted literally and not as an escape. I have another special character that separates the "header" from the "message" and the last 2 bytes of message are the checksum.

I do in fact need to escape my sync bytes because I don't have any other way of detecting the starts or ends of transmissions besides my special flag bytes.

The real trouble is when you need complete reliability and you need to be able to automatically resend if the listener doesn't get the packet. But this doesn't really come up very often in hobby stuff. What I do is include an incrementing in the initial packet that gets echoed back by the listener, the sender will resend if it doesn't get the response, and the listener will ignore duplicates based on the counter.

The other way is the "pull" method where the listener requests a piece of data and repeats the request if needed.

You really do want a checksum. I like the modified Fletcher checksum where you take the mod 256 sum of all the bytes, and you also take the sum of that sum value at every step. Hard to explain but the code is like four lines. I stayed away from CRCs because I wanted the minimum resource usage, but CRCs are more reliable.

Really,there won't be very many errors, so unless every single packet is extremely important(one missed temperature reading an hour out of thousands isn't a big deal to most people), the only thing to worry about is detecting errors(because a bad packet can be worse than a dropped packet) and synchronization errors(because you don't want one bad sequence to cause a locked up state)

That's one of the most important things in protocol design: make sure there is no sequence of data on the wire capable of causing the decoder to enter into an infinite wait period.

A lot of people like to base their packet structure on a length byte near the start of the packet. I'm not much of a fan, partly because IMHO the code to check for an undisclosed ending newline is a lot cleaner looking that "does the total length match the length byte, and have we gotten far enough along in the packet to have actually recieved a length byte. Also, if you do multi master busses with collision detection, start and stop bytes allow instant collision recovery as a start of packet char immediately discards any incomplete data sitting around with no waiting for timeouts.

Some protocols dellimit packets by empty space of a certain length. This would be my personal least favorite because it adds a fairly strict timing dependance.