Reading UART serial data without loosing bytes

Projects Programming
1

I need to read and respond to serial data being communicated at 19200bps. The data being transmitted are hex bytes that form commands (Apple Accessory Protocol Commands).

For example:
0xFF 0x55 0x04 0x04 0x00 0x26 0x01 0xD1 is a command to play or pause the current song.

It is formatted as follows:
0xFF 0x55 -> standard header
0x04 -> length of command mode+command+parameters (4 bytes long not including checksum)
0x04 -> command mode (mode 4, advanced control mode. 1 byte)
0x00 0x26 -> command (media control. 2 bytes)
0x01 -> parameter (play. 1 byte)
0xD1 -> checksum (1 byte)

The length of the commands can range from 7 bytes to 512 bytes and sometimes there are garbage bytes that get send.
ie: 0xFF 0x00 0xFF 0xFF 0x55 0x04 0x04 0x00 0x26 0x01 0xD1
where the first 3 bytes need to be ignored.

Most of the time, only one command is being sent at a time. This makes it fairly simple. However, there are times where multiple commands are sent at once, which means that my system has to parse out the commands and decode them one by one to send the appropriate responses. When multiple commands are sent, somewhere around 128 bytes (composed of ~15 commands) are sent.

I have tried reading the bytes one at a time, but on large commands or multiple commands, i end up loosing bytes in the Rx buffer. I switched gears and tried the Serial.readBytes() function. It works better, but I am confused how the Serial.setTimeout() method works. I tried:

char buffer[512];
...
Serial.setTimeout(10);
...
Serial.readBytes(buffer, 512);

Now, does this mean that Serial.readBytes will read bytes into buffer for 10ms OR if 512 bytes are read? Or does it mean that it will read up to 512 bytes into buffer as long as data bytes are being received at lease every 10ms, so that a timeout occurs if there is no Rx receives for 10ms or longer?

Is there a better way to do this in general? My understanding is that Arduino 1.0 and greater utilizes interrupt driven UART, but the Rx buffer is only 64 bytes. this means that I need to be able to pull out all the bytes in a command from the UART buffer, decode the command, generate and send a response very quickly for large commands or if there are multiple commands to prevent data loss. I believe the 8Mhz Arduino I am using can process fast enough to accomplish this, but it looks like there is a great deal of overhead with reading bytes one at a time using Serial.read() and using the polling method ( while(Serial.available() < 7); ) to ensure im reading from the hardware Rx buffer only when data is available.

I guess I can always increase the hardware Rx buffer to 512 bytes by modifying the hardware serial library (which will use 1Kb of program memory for Rx and Tx, but that's OK for my application as its not that big to begin with and i have 16K to play with).

I was hopping for some ideas for a better implementation where i can read one or multiple commands and process them (to remove them from the stock 64 byte buffer) without losing data.

I appreciate the help!

2

while(Serial.available() < 7); That won't work. What will happen when there are zero bytes available to read ? Surely what you want to do is to read a byte whenever it is available and process it.

Wait until you get 0xFF 0x55 in consecutive reads, assuming that the combination cannot occur in the data, read the next byte to get the message length, read the message into an array, read the checksum and check it if you want to. Act on the message. Go back and do it again.

You say that you have tried this but lost bytes. What were you doing between reading bytes that took so long ?

Incidentally, I would strongly suspect that the "garbage" bytes that you get are no such thing, just that you do not know what they mean.

From the Arduino reference page

Serial.setTimeout() sets the maximum milliseconds to wait for serial data when using Serial.readBytesUntil() or Serial.readBytes(). It defaults to 1000 milliseconds.

3

Sounds like a job for a simple state machine...
State 1 = Look for FF, if found goto state 2 else stay at state 1.
State 2 = is next value 55? If yes then goto state 3 else back to state 1. Back to state 1 if timeout.
State 3 = read number of bytes for command and goto state 4 or back to state 1 if timeout.
State 4 = read x data bytes then goto state 5 else state 1 if timeout.
State 5 = read and compare CRC if valid goto state 6 else goto state 1
State 6 = act on received command (and acknowledge it?). goto state 1

4

I guess I can always increase the hardware Rx buffer to 512 bytes by modifying the hardware serial library (which will use 1Kb of program memory for Rx and Tx, but that's OK for my application as its not that big to begin with and i have 16K to play with).

Guess again. The buffers are in SRAM, not program memory. You do NOT have 16K to play with.

Serial data arrives ssslllooowwwlllyyy. The Arduino can read it quite fast, if you quit farting around, doing whatever you're doing, and get back to reading data sooner. Increasing the buffer size is rarely the answer.

5

There seems to be a duplicated, redundant 'o' in your serial data.

6

Wow, so many items to reply to! Thank you for the feedback!
Let's get started...

UKHeliBob:
while(Serial.available() < 7); That won't work. What will happen when there are zero bytes available to read ? Surely what you want to do is to read a byte whenever it is available and process it.

You say that you have tried this but lost bytes. What were you doing between reading bytes that took so long ?

Incidentally, I would strongly suspect that the "garbage" bytes that you get are no such thing, just that you do not know what they mean.

From the Arduino reference page

Serial.setTimeout() sets the maximum milliseconds to wait for serial data when using Serial.readBytesUntil() or Serial.readBytes(). It defaults to 1000 milliseconds.

Actually, I misspoke. The while (Serial.available() < 7); code is only used if we are sure we are in a command (by finding a 0xFF and 0x55 together). Then the spinlock on the serial availability is used to ensure a minimum byte count is available before attempting to read again.

  //process command if we have more than 6 bytes available
  if(Serial.available() > 6) parseCommand();
...
[IN PARSECOMMAND FUNCTION]
  //find start of the command. ignore garbage bytes in the beginning 
  while(!((Serial.read() == 0xFF) && (Serial.peek() == 0x55)))
  {
    if(!Serial.available()) return;
  }
  Serial.read(); //read the 0x55
  uint8_t i = 0;
  while(Serial.available() < 1);
  uint8_t cmdLength = Serial.read();

I read the 2 sentences in the Arduino page regarding the setTimeout method. It s still not clear what exactly is meant. Does the timeout occur if there is more time "between bytes" than the timeout parameter, or does that mean that methods like readBytes will only read for that duration, regardless if the number of bytes defined to read has been read?

Riva:
Sounds like a job for a simple state machine...
State 1 = Look for FF, if found goto state 2 else stay at state 1.
State 2 = is next value 55? If yes then goto state 3 else back to state 1. Back to state 1 if timeout.
State 3 = read number of bytes for command and goto state 4 or back to state 1 if timeout.
State 4 = read x data bytes then goto state 5 else state 1 if timeout.
State 5 = read and compare CRC if valid goto state 6 else goto state 1
State 6 = act on received command (and acknowledge it?). goto state 1

This is close to what I am doing. I have a subroutine that searches for the header in the Rx buffer if data is available:

  while(!((Serial.read() == 0xFF) && (Serial.peek() == 0x55)))
  {
    if(!Serial.available()) return;
  }

The it waits until the length byte is available and reads it. This tells it how many more bytes it must wait for to become available. before reading them.

PaulS:

I guess I can always increase the hardware Rx buffer to 512 bytes by modifying the hardware serial library (which will use 1Kb of program memory for Rx and Tx, but that's OK for my application as its not that big to begin with and i have 16K to play with).

Guess again. The buffers are in SRAM, not program memory. You do NOT have 16K to play with.

Serial data arrives ssslllooowwwlllyyy. The Arduino can read it quite fast, if you quit farting around, doing whatever you're doing, and get back to reading data sooner. Increasing the buffer size is rarely the answer.

I realized that i was trying to read data more quickly than it was available and that made me think that I was loosing bytes. In fact, I was calling Serial.read before the next byte made it there. However, I do not believe that we are talking about the same buffers. The buffer I am referring to is the ring buffer that data gets placed in from the Rx receive interrupt. It is an array defined in HardwareSerial.cpp

#if (RAMEND < 1000)
  #define SERIAL_BUFFER_SIZE 16
#else
  #define SERIAL_BUFFER_SIZE 64 //Using this one
#endif

struct ring_buffer
{
  unsigned char buffer[SERIAL_BUFFER_SIZE]; 
  volatile unsigned int head;
  volatile unsigned int tail;
};

#if defined(USBCON)
  ring_buffer rx_buffer = { { 0 }, 0, 0}; // No dynamic allocation. Not in RAM!!
  ring_buffer tx_buffer = { { 0 }, 0, 0};
#endif

Then, in the UART0 ISR, a byte is read and stored in the buffer:

      unsigned char c = UDR0;
      store_char(c, &rx_buffer);

This is all program memory, right? i dont see any heap usage here. If I increase

#define SERIAL_BUFFER_SIZE 64
to 
#define SERIAL_BUFFER_SIZE 512

I am using more program memory (which my compile size confirms) which is fine as my project is not large otherwise.

7

The buffers are in ram. You cannot (normally) write to program memory during runtime.