Can't Avoid Serial Buffer Overflow!

Hello all,

I am writing a program which essentially collects data from external sensors/boards and then sends it out to a laptop with an XBee dongle. Currently I have a GPS module, an IMU, a digital temperature sensor, an XBee, an RS232 serial shifter, and an OpenLog (all hooked up to an Arduino MEGA 2560). I collect data from the GPS, IMU, temperature sensor, and the serial shifter and send it all out over the XBee (and to the OpenLog). The issue I am having is the device sending data to the serial shifter sends a constant stream of data, no pauses. The device does not allow any sort of hardware flow control or handshaking of any sort. The device sends 25-byte packets of data one after the other at 9600bps. My first approach to parse the packet was to collect sample data, say 50-bytes, then search for the three start delimiters (0x10, 0x10, 0x10) in the 25-byte packet and then just send the 25-byte packet out. The issue I am having is I get some serious buffer overflow. I have tweaked the delay after I flush the Serial 3 buffer and I have found that a delay of 42ms produces the least amount of errors in my data, however, I am still receiving tons of errors due to the invalid data caused by the overflow.

How can I prevent the overflow and maintain data reliability? The data that is sent out from the Arduino to the laptop is important and the highest possible accuracy is crucial.

Thank you for your help!!!

#include <Wire.h>

float farenheit;
int temp;
int val;
float convertedtemp;
int i;
int j;
byte firstbyte;
byte secondbyte;
byte buffer[50];

void setup() 
{
  Wire.begin();
  Serial.begin(9600); // Computer serial port (to be used for OpenLog later)
  Serial1.begin(115200); // GPS Serial Port
  Serial2.begin(57600); // IMU serial port
  Serial3.begin(9600); // XBee/Serial Shifter serial port
}

void loop() 
{ 
  /**************************************************************
  The following code is for the GPS module. The baud rate is set
  to 115,200bps in the firmware and it polls at 2Hz. It will
  only print the GPRMC statement.
  ***************************************************************/
  while (Serial1.available() > 0) // Check if data exists in Serial 1 read buffer, if it does...
  {
    temp = Serial1.read(); // Read and store byte into temp variable
    if (temp != 10) // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial1.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  /**************************************************************
  The following code is for the IMU. The baud rate is set to 
  57,600bps in the firmware and it will print the following:
  $ANG,Roll,Pitch,Yaw
  ***************************************************************/
  while (Serial2.available() > 0) // Check if data exists in Serial 2 read buffer, if it does...
  {
    temp = Serial2.read(); // Read and store byte into temp variable
    if (temp != 10)  // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial2.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  Serial2.write(1); // Sends handshake to IMU to let it know that we are ready for data
  // delay(20);
  
   /***************************************************************
  The following code is for the serial shifter which receives data
  from the battery management system. It will print data serially
  in the following format at 9600bps: $BMS,Data,Data,Data....
  ***************************************************************/
  if (Serial3.available() > 0) // If there is data in the Serial 3 read buffer...
  {
    i = 0; // Reset counter
    Serial3.flush(); // Flush the Serial3 read buffer so it can fill with new data
    delay(42); // Delay to give time for Serial3 read buffer to fill
    while(i <= 50){ // Fill buffer with 50 elements
       buffer[i] = Serial3.read(); // Read byte from serial buffer and store in our buffer
       i++; // Increment counter
    }
    for(j = 0; j <= 50; j++) // A for loop which will traverse our buffer
    {
      if(buffer[j] == 0x10) break; // Once first start delimiter is reached, break from for statement
      temp = j; // Store position of start delimiter in variable
    }
    if((buffer[j+1] == 0x10) && (buffer[j+2] == 0x10)) // If the next two start delimiters are present in the buffer...
    { 
      Serial3.print("$BMS"); // Print our start delimiter to the XBee
      Serial.print("$BMS"); // Print our start delimiter to the OpenLog
      for(j; j<=(temp + 25); j++) // A for statement that will only print the 25 bytes from the packet
      {
        if(j != (temp + 25)){ // While we aren't on the last element in the packet...
          Serial3.print(","); // Print comma delimiter to XBee
          Serial.print(","); // Print comma delimiter to OpenLog
          Serial3.print((int)buffer[j]); // Print current buffer element to XBee
          Serial.print((int)buffer[j]); // Print current buffer element to OpenLog
        }
       else { Serial3.println(); Serial.println(); } // Send a new line at the end of a packet
    } 
    }
  }
 
  /***************************************************************
  The following code is for the TMP-102 digital temperature sensor
  and two potentiometers (acceleration and develeration).
  It will print the following: $MOTOR,Temp,Accel,Decel
  ***************************************************************/
  Wire.beginTransmission(72); // Start transmission with TMP-102 sensor
  Wire.send(0x00); // Send two zero bytes to poll sensor for data
  Wire.endTransmission(); // End send transmission with TMP-102 sensor
  Wire.requestFrom(72, 2); // Request two bytes from sensor
  Wire.endTransmission(); // End request trasmission with TMP-102 sensor
  
  firstbyte = Wire.receive(); // Receive first byte and store
  secondbyte = Wire.receive(); // Receive second byte and store
  
  val = ((firstbyte) << 4); // Shift first byte to the left
  val |= (secondbyte >> 4); // Shift second byte to to the right and OR with val

  farenheit = ((1.8 * (val * 0.0625)) + 32) - 5; // Convert to farenheit and subtract since reads over by 5 degrees
  
  int accel = analogRead(A0); // Read acceleration potentiometer, convert to percentage, and store in accel variable
  if (accel > 1000) accel = 1000; // If the accel value is greater than 1000, truncate it to 1000
  int decel = analogRead(A1); // Read decerleation potentiometer, convert to percentage, and store in decel variable
  if (decel > 1000) decel = 1000; // If the decel value is greater than 1000, truncate it to 1000

  Serial3.print("$MOTOR,"); // Print beginning of MOTOR line to the XBee
  Serial3.print(farenheit); // Print temperature to XBee
  Serial3.print(","); // Insert comma delimiter
  Serial3.print(accel / 10); // Print accel in percentage
  Serial3.print(","); // Insert comma delimiter
  Serial3.println(decel / 10); // Print decel in percentage and insert a new line
  
  Serial.print("$MOTOR,"); // Print beginning of MOTOR line to the OpenLog
  Serial.print(farenheit); // Print temperature to OpenLog
  Serial.print(","); // Insert comma delimiter
  Serial.print(accel / 10); // Print accel in percentage
  Serial.print(","); // Insert comma delimiter
  Serial.println(decel / 10); // Print decel in percentage and insert a new line
}

you should check if you are not producing more data than you can send per second over the XBEE line to the PC.

When your XBEE can transmit 9600 bits per second ~ 1000 bytes and your sensor produce more, then you have a problem.

Solution is to compactify the data to its bare essentials

  • removing anything not significant (like strings that are constant, digits you do not use etc)
  • compress the remaining data.

if that is not enough you should lower the frequency of your readings.

The XBee is capable of sending data far faster than you are reading it from the dumdass device. If you set the same speed for the output as for the input, you will eventually (quickly) fall behind. If you can send out 12 times as fast as you get input (12 * 9600 = 115200, which the XBee can do), you have far less risk of falling behind.

Thanks for the input! I previously attempted to set the Interface Data Rate on both XBees to 115200bps and I got a lot of noise in the data, the 9600bps rate was the most reliable. So you think the solution to my issue would be to increase the serial transfer rate on the XBees to the highest possible rate?

EDIT: I should note that I have been testing the board sans the XBee and just using the USB (Serial0) for debugging. The issue is still quite prevalent when using a baud rate of 115200bps and reading over USB.

The issue I am having is the device sending data to the serial shifter sends a constant stream of data, no pauses. The device does not allow any sort of hardware flow control or handshaking of any sort. The device sends 25-byte packets of data one after the other at 9600bps.

Is there a lot of variation in the data, or does that device send pretty much the same information time after time?

If there is a lot of variation, then it is probably critical that you capture it all. If there is little variation, then missing a packer or 12 now and then probably doesn't matter.

If there is little variation, simply flush the buffer, and then discard data up to the next start of packet marker. Store the data from up to the end of packet marker, then process it and repeat.

PaulS:
Is there a lot of variation in the data, or does that device send pretty much the same information time after time?

If there is a lot of variation, then it is probably critical that you capture it all. If there is little variation, then missing a packer or 12 now and then probably doesn't matter.

If there is little variation, simply flush the buffer, and then discard data up to the next start of packet marker. Store the data from up to the end of packet marker, then process it and repeat.

At the current processing speed the data variation is pretty small. I inserted a small 5ms delay in the print loop and that has somewhat solved my issue. I no longer get the large amounts of buffer overflow, but I now get weird values. So the packet structure is: 0x10 0x10 0x10 0xXX 0xXX 0xXX so on and so forth for 25 bytes. For some reason, after the three start delimiters I am getting 255 for most (if not all) of the values. However, I wrote a small read-and-dump program which just reads the byte and sends it out over the XBee. That simple read-and-dump works perfectly and the values are all valid, but when I swap the read-and-dump for the program above, I get 0x10 0x10 0x10 0x00 0x00 0xFF 0xFF..... Confused.

If you're getting a lot of 0xFF on your serial read, it suggests that there's nothing there - the read is returning -1. In this section:

    Serial3.flush(); // Flush the Serial3 read buffer so it can fill with new data
    delay(42); // Delay to give time for Serial3 read buffer to fill
    while(i <= 50){ // Fill buffer with 50 elements
       buffer[i] = Serial3.read(); // Read byte from serial buffer and store in our buffer

You are assuming that 42 ms will guarantee you 50 bytes to read. At 9600 baud, assuming 1 start, 2 stop, no parity, that's only enough time for 40, and even then that would require that they're being put on the wire with no time expended between them. Try using Serial.Available as you have elsewhere in the code to ensure that there's something to read.

wildbill:
If you're getting a lot of 0xFF on your serial read, it suggests that there's nothing there - the read is returning -1. In this section:

    Serial3.flush(); // Flush the Serial3 read buffer so it can fill with new data

delay(42); // Delay to give time for Serial3 read buffer to fill
    while(i <= 50){ // Fill buffer with 50 elements
       buffer[i] = Serial3.read(); // Read byte from serial buffer and store in our buffer




You are assuming that 42 ms will guarantee you 50 bytes to read. At 9600 baud, assuming 1 start, 2 stop, no parity, that's only enough time for 40, and even then that would require that they're being put on the wire with no time expended between them. Try using Serial.Available as you have elsewhere in the code to ensure that there's something to read.

I can't believe I didn't pick up on the -1 for Serial3.read(), I feel like an idiot. I have modified my code to check for Serial3.available >= 50 and I have kept the 5ms delays in the print statements.... Unfortunately I am unable to check this code until tomorrow, but in theory this should prevent buffer overflow and my empty serial buffer issue, right?

#include <Wire.h>

float farenheit;
int temp;
int val;
float convertedtemp;
int i;
int j;
byte firstbyte;
byte secondbyte;
byte buffer[50];

void setup() 
{
  Wire.begin();
  Serial.begin(9600); // Computer serial port (to be used for OpenLog later)
  Serial1.begin(115200); // GPS Serial Port
  Serial2.begin(57600); // IMU serial port
  Serial3.begin(9600); // XBee/Serial Shifter serial port
}

void loop() 
{ 
  /**************************************************************
  The following code is for the GPS module. The baud rate is set
  to 115,200bps in the firmware and it polls at 2Hz. It will
  only print the GPRMC statement.
  ***************************************************************/
  while (Serial1.available() > 0) // Check if data exists in Serial 1 read buffer, if it does...
  {
    temp = Serial1.read(); // Read and store byte into temp variable
    if (temp != 10) // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial1.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  /**************************************************************
  The following code is for the IMU. The baud rate is set to 
  57,600bps in the firmware and it will print the following:
  $ANG,Roll,Pitch,Yaw
  ***************************************************************/
  while (Serial2.available() > 0) // Check if data exists in Serial 2 read buffer, if it does...
  {
    temp = Serial2.read(); // Read and store byte into temp variable
    if (temp != 10)  // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial2.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  Serial2.write(1); // Sends handshake to IMU to let it know that we are ready for data
  
   /***************************************************************
  The following code is for the serial shifter which receives data
  from the battery management system. It will print data serially
  in the following format at 9600bps: $BMS,Data,Data,Data....
  ***************************************************************/
  if (Serial3.available() >= 50) // If there is at least 50 bytes in the Serial 3 buffer...
  {
    i = 0; // Reset counter
    while(i <= 50){ // Fill buffer with 50 elements
       buffer[i] = Serial3.read(); // Read byte from serial buffer and store in our buffer
       i++; // Increment counter
    }
    for(j = 0; j <= 50; j++) // A for loop which will traverse our buffer
    {
      if(buffer[j] == 0x10) break; // Once first start delimiter is reached, break from for statement
      temp = j; // Store position of start delimiter in variable
    }
    if((buffer[j+1] == 0x10) && (buffer[j+2] == 0x10)) // If the next two start delimiters are present in the buffer...
    { 
      Serial3.print("$BMS"); // Print our start delimiter to the XBee
      Serial.print("$BMS"); // Print our start delimiter to the OpenLog
      for(j; j<=(temp + 25); j++) // A for statement that will only print the 25 bytes from the packet
      {
        if(j != (temp + 25)){ // While we aren't on the last element in the packet...
          Serial3.print(","); // Print comma delimiter to XBee
          Serial.print(","); // Print comma delimiter to OpenLog
          delay(5);
          Serial3.print((int)buffer[j]); // Print current buffer element to XBee
          Serial.print((int)buffer[j]); // Print current buffer element to OpenLog
          delay(5);
        }
       else { Serial3.println(); Serial.println(); } // Send a new line at the end of a packet
    } 
    }
    Serial3.flush(); // Clear out the buffer to make space for new data
  }
 
  /***************************************************************
  The following code is for the TMP-102 digital temperature sensor
  and two potentiometers (acceleration and develeration).
  It will print the following: $MOTOR,Temp,Accel,Decel
  ***************************************************************/
  Wire.beginTransmission(72); // Start transmission with TMP-102 sensor
  Wire.send(0x00); // Send two zero bytes to poll sensor for data
  Wire.endTransmission(); // End send transmission with TMP-102 sensor
  Wire.requestFrom(72, 2); // Request two bytes from sensor
  Wire.endTransmission(); // End request trasmission with TMP-102 sensor
  
  firstbyte = Wire.receive(); // Receive first byte and store
  secondbyte = Wire.receive(); // Receive second byte and store
  
  val = ((firstbyte) << 4); // Shift first byte to the left
  val |= (secondbyte >> 4); // Shift second byte to to the right and OR with val

  farenheit = ((1.8 * (val * 0.0625)) + 32) - 5; // Convert to farenheit and subtract since reads over by 5 degrees
  
  int accel = analogRead(A0); // Read acceleration potentiometer, convert to percentage, and store in accel variable
  if (accel > 1000) accel = 1000; // If the accel value is greater than 1000, truncate it to 1000
  int decel = analogRead(A1); // Read decerleation potentiometer, convert to percentage, and store in decel variable
  if (decel > 1000) decel = 1000; // If the decel value is greater than 1000, truncate it to 1000

  Serial3.print("$MOTOR,"); // Print beginning of MOTOR line to the XBee
  Serial3.print(farenheit); // Print temperature to XBee
  Serial3.print(","); // Insert comma delimiter
  Serial3.print(accel / 10); // Print accel in percentage
  Serial3.print(","); // Insert comma delimiter
  Serial3.println(decel / 10); // Print decel in percentage and insert a new line
  
  Serial.print("$MOTOR,"); // Print beginning of MOTOR line to the OpenLog
  Serial.print(farenheit); // Print temperature to OpenLog
  Serial.print(","); // Insert comma delimiter
  Serial.print(accel / 10); // Print accel in percentage
  Serial.print(","); // Insert comma delimiter
  Serial.println(decel / 10); // Print decel in percentage and insert a new line
}

Thank you so much for your help, you guys have saved me so much debugging time!!!

RyanC:
I feel like an idiot.

Get used to it. It happens to me every time I debug my code :wink:

RyanC:
but in theory this should prevent buffer overflow and my empty serial buffer issue, right?

Probably. However, I would be more inclined to read characters to the buffer whenever they present themselves and process them when you have 50. That way, the other portions of loop() will get serviced more often.

So with the new code (below) I can successfully read all data and parse it as necessary. The newest issue is that the data reads fine on the computer-side for a few minutes but then eventually starts to slow down and eventually becoming severely behind. The application requires low latency as it needs the latest GPS and IMU data as soon as possible. Is there any way to combat the high latency of the Serial.print method?

#include <Wire.h>

float farenheit;
int temp;
int val;
float convertedtemp;
int i;
int j;
byte firstbyte;
byte secondbyte;
byte buffer[50];

void setup() 
{
  Wire.begin();
  Serial.begin(9600); // Computer serial port (to be used for OpenLog later)
  Serial1.begin(115200); // GPS Serial Port
  Serial2.begin(57600); // IMU serial port
  Serial3.begin(9600); // XBee/Serial Shifter serial port
}

void loop() 
{ 
  /**************************************************************
  The following code is for the GPS module. The baud rate is set
  to 115,200bps in the firmware and it polls at 2Hz. It will
  only print the GPRMC statement.
  ***************************************************************/
  while (Serial1.available() > 0) // Check if data exists in Serial 1 read buffer, if it does...
  {
    temp = Serial1.read(); // Read and store byte into temp variable
    if (temp != 10) // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial1.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  /**************************************************************
  The following code is for the IMU. The baud rate is set to 
  57,600bps in the firmware and it will print the following:
  $ANG,Roll,Pitch,Yaw
  ***************************************************************/
  while (Serial2.available() > 0) // Check if data exists in Serial 2 read buffer, if it does...
  {
    temp = Serial2.read(); // Read and store byte into temp variable
    if (temp != 10)  // If received character is NOT a new line character...
    {
      Serial3.print(temp, BYTE); // Print the byte to the XBee
      Serial.print(temp, BYTE); // Print the byte to the OpenLog
    }
    else // If received character IS a new line character...
    {
      Serial3.println(); //Insert a new line
      Serial.println(); // Insert a new line
      Serial2.flush(); // Just in case
      break; // Break out of the while loop
    }
  }
  Serial2.write(1); // Sends handshake to IMU to let it know that we are ready for data
  
   /***************************************************************
  The following code is for the serial shifter which receives data
  from the battery management system. It will print data serially
  in the following format at 9600bps: $BMS,Data,Data,Data....
  ***************************************************************/
  if (Serial3.available() >= 50) // If there is at least 50 bytes in the Serial 3 buffer...
  {
    i = 0; // Reset counter
    while(i <= 50) // Fill buffer with 50 elements
    {
       buffer[i] = Serial3.read(); // Read byte from serial buffer and store in our buffer
       i++; // Increment counter
    }
    for(j = 0; j <= 50; j++) // A for loop which will traverse our buffer
    {
      if(buffer[j] == 0x10) // If first start delimiter is found...
      {
        temp = j; // Store position of start delimiter in variable
        break; // Break from for statement
      }
    }
    if((buffer[j+1] == 0x10) && (buffer[j+2] == 0x10)) // If the next two start delimiters are present in the buffer...
    { 
      Serial3.print("$BMS"); // Print our start delimiter to the XBee
      Serial.print("$BMS"); // Print our start delimiter to the OpenLog
      for(j; j<=(temp + 25); j++) // A for statement that will only print the 25 bytes from the packet
      {
        if(j != (temp + 25)) // While we aren't on the last element in the packet...
        {
          Serial3.print(","); // Print comma delimiter to XBee
          Serial.print(","); // Print comma delimiter to OpenLog
          Serial3.print((int)buffer[j]); // Print current buffer element to XBee
          Serial.print((int)buffer[j]); // Print current buffer element to OpenLog
        }
        else 
        { 
          Serial3.println(); // Send a new line at the end of a packet
          Serial.println(); 
        }
     } 
    }
    Serial3.flush(); // Clear out the buffer to make space for new data
  }
 
  /***************************************************************
  The following code is for the TMP-102 digital temperature sensor
  and two potentiometers (acceleration and develeration).
  It will print the following: $MOTOR,Temp,Accel,Decel
  ***************************************************************/
  Wire.beginTransmission(72); // Start transmission with TMP-102 sensor
  Wire.send(0x00); // Send two zero bytes to poll sensor for data
  Wire.endTransmission(); // End send transmission with TMP-102 sensor
  Wire.requestFrom(72, 2); // Request two bytes from sensor
  Wire.endTransmission(); // End request trasmission with TMP-102 sensor
  
  firstbyte = Wire.receive(); // Receive first byte and store
  secondbyte = Wire.receive(); // Receive second byte and store
  
  val = ((firstbyte) << 4); // Shift first byte to the left
  val |= (secondbyte >> 4); // Shift second byte to to the right and OR with val

  farenheit = ((1.8 * (val * 0.0625)) + 32) - 5; // Convert to farenheit and subtract since reads over by 5 degrees
  
  int accel = analogRead(A0); // Read acceleration potentiometer, convert to percentage, and store in accel variable
  if (accel > 1000) accel = 1000; // If the accel value is greater than 1000, truncate it to 1000
  int decel = analogRead(A1); // Read decerleation potentiometer, convert to percentage, and store in decel variable
  if (decel > 1000) decel = 1000; // If the decel value is greater than 1000, truncate it to 1000

  Serial3.print("$MOTOR,"); // Print beginning of MOTOR line to the XBee
  Serial3.print(farenheit); // Print temperature to XBee
  Serial3.print(","); // Insert comma delimiter
  Serial3.print(accel / 10); // Print accel in percentage
  Serial3.print(","); // Insert comma delimiter
  Serial3.println(decel / 10); // Print decel in percentage and insert a new line
  
  Serial.print("$MOTOR,"); // Print beginning of MOTOR line to the OpenLog
  Serial.print(farenheit); // Print temperature to OpenLog
  Serial.print(","); // Insert comma delimiter
  Serial.print(accel / 10); // Print accel in percentage
  Serial.print(","); // Insert comma delimiter
  Serial.println(decel / 10); // Print decel in percentage and insert a new line
}

Is there any way to combat the high latency of the Serial.print method?

  Serial.begin(9600); // Computer serial port (to be used for OpenLog later)

Get a move on. You can print 12 times as fast as that.

PaulS:
Get a move on. You can print 12 times as fast as that.

Well my Xbee is being stubborn and is giving me lots of problems at 115200bps, but when I tested the program at 115200bps via USB (Serial0) I still experienced that lag albeit a little later than when the baud rate is 9600bps. If I let the program run for 10-15mins it can get up to 4mins behind!!! That's a lot!

I can't believe that i was not knowing this!!