Dealing with buffer overflow

Hi there,

I'm developing an Arduino to receive serial data from another device at 19200 and I don't control how often this device sends the data, assume that it might send data constantly and indefinitely.

I process this data and send it to a computer using the USB connection (Touch Screen HID) using an Arduino Micro Pro, here's the code:

#include "Touch.h"

const int FINGER = 1;

int i;
byte touch_data[6];
bool state_pressed = false;

void setup() {
  Touch.begin();
  Serial.begin(19200);
}

void loop() {
  if (Serial.available() > 0) {
    // Capture 6 bytes
    for (i = 0; i < 6; i++) {
      touch_data[i] = Serial.read();
    }

    // 1 byte const
    // 1 byte const
    // 1 byte + 1 byte x
    // 1 byte + 1 byte y
    int16_t x = word(touch_data[2], touch_data[3]);
    int16_t y = word(touch_data[4], touch_data[5]);

    // Adjust from 0..65535 to 0..10000
    x = (x * 10000) / 65535;
    y = (y * 10000) / 65535;

    Touch.moveFingerTo(FINGER, x, y);

    state_pressed = true;
  } else if (state_pressed) {
    Touch.releaseFinger(FINGER);
    state_pressed = false;
  }
}

As you can see, the device sends 6 bytes and I care about the last four, the first two are constant values.

What happens if the computer takes a while to receive the computed data from Arduino and the device sends hundreds of messages while that?

There is any kind of overflow situation that I need to handle?

I'm fine with losing/ignoring data to recover from that if needed.

Thanks! <3

you need to check each byte if available(), not read 6 if one available

How is this other device connected to the Pro Micro? Using pins 0, 1? If so, you need to be using Serial1, not Serial (which is the USB connection to the pc).

#include "Touch.h"

const int FINGER = 1;

int i;
byte touch_data[5];
bool state_pressed = false;

void setup() {
  Touch.begin();
  Serial1.begin(19200);
}

void loop() {
  if (Serial1.available() == 6) {
    // Capture 6 bytes
    for (i = 0; i <= 5; i++) {
      touch_data[i] = Serial1.read();
    }

    // 1 byte const
    // 1 byte const
    // 1 byte + 1 byte x
    // 1 byte + 1 byte y
    int16_t x = word(touch_data[2], touch_data[3]);
    int16_t y = word(touch_data[4], touch_data[5]);

    // Adjust from 0..65535 to 0..10000
    x = (x * 10000) / 65535;
    y = (y * 10000) / 65535;

    Touch.moveFingerTo(FINGER, x, y);

    state_pressed = true;
  } else if (state_pressed) {
    Touch.releaseFinger(FINGER);
    state_pressed = false;
  }
}

So, like that?

Yeah, it will be connected on pins 0 and 1.

I'm fine with losing/ignoring data to recover from that if needed.

Rather than ignoring those 2 bytes with constant values, use them to re-synchronise the incoming data, in case some data has been missed/lost.

Suppose those two constant values are A and B. Rather than reading 6 bytes at once (which might not start with A, B if some data has been lost) read 1 byte at a time, when Serial1.available() > 0. Use a couple of flag variables to keep track of things, like 'receivedA' and 'receivedB'. If A has not been received yet and the newly received byte is A, then set the flag for A. If A has been received, and the newly received byte is B, then set the flag for B. But if that byte is not B, then what we thought was A was just a fluke: clear both flags. If the flag for B is set, we must have received both A and B, and are synchronised with the incoming data, so the next 4 bytes to be received can be used safely as data.

If that sounds difficult, another way would be to read one byte at a time, but always write it to index 5 of your array, having first moved the other bytes one place to the left (ie. index 1 overwrites index 0, index 2 overwrites index 1 etc). Then, if index 0 is A and index 1 is B, you can safely use indexes 2 to 5 as your data.

So, like that?

Why did you change this line? 6 was correct.

byte touch_data[5];

It was a typo, was playing around and changed the wrong line.

Is that possible to receive const A, const B, and the wrong data in the next two bytes?

I for sure can discard the data if it's not like A,B,x,y but if I receive A,B does it mean for sure that the next two bytes are what I originally transmitted?

No, you are correct, it's not a 100% guarantee. In theory, it's a 1:64,000 chance of being wrong, or a 99.998% chance of being right. That's the best you can do.

the next two bytes are what I originally transmitted?

I thought you were not in control of the bytes transmitted? If you are, why not send 3 or 4 bytes of constant values, to reduce the tiny chance of being wrong.

I'm not in control, I expressed myself in the wrong way! :slight_smile:

The question would be if the last 4 bytes would be reliable if the first 2 bytes are what I expected them to be.

98-99% accuracy is quite good here.

Now, what about overflow? Do I need to do something about that or that doesn't even happen?

#include "Touch.h"

const int FINGER = 1;

int i;
byte touch_data[6];
uint16_t x;
uint16_t y;
bool state_pressed = false;

void setup() {
  Touch.begin();
  Serial1.begin(19200);
}

void loop() {
  if (Serial1.available() > 0) {
    // Capture 6 bytes
    for (i = 0; i <= 5; i++) {
      touch_data[i] = Serial1.read();
    }

    // Detect out of sync messages, protection against buffer overflow
    if (touch_data[0] != 0xa1 || touch_data[1] != 0x00) {
      return;
    }

    // 1 byte const
    // 1 byte const
    // 1 byte + 1 byte x
    // 1 byte + 1 byte y
    x = word(touch_data[2], touch_data[3]);
    y = word(touch_data[4], touch_data[5]);

    // Adjust from 0..65535 to 0..10000
    x = (x / 65535.0) * 10000.0;
    y = (y / 65535.0) * 10000.0;

    Touch.moveFingerTo(FINGER, x, y);

    state_pressed = true;
  } else if (state_pressed) {
    Touch.releaseFinger(FINGER);
    state_pressed = false;
  }
}

Looks like it is working, just wondering about the overflow, if there are good practices about that now.

Still working on it but almost there:

#include "Touch.h"

const int FINGER = 1;
const int PACKET_SIZE = 6;

byte packet[PACKET_SIZE];
uint16_t x;
uint16_t y;
bool isMoving = false;

void setup() {
  Touch.begin();

  Serial1.begin(19200);
  Serial1.setTimeout(5); // Qual o tempo para receber após o primeiro byte?
}

void loop() {
  if (Serial1.available() > 0) {
    int size = Serial1.readBytes(packet, PACKET_SIZE);

    // Ignore if the entire packet couldn't be read.
    if (size != PACKET_SIZE) {
      return;
    }

    // In the case of corrupted data, ignore and try again.
    //
    //   byte 0 = 0xa1
    //   byte 1 = 0x00
    if (packet[0] != 0xa1 || packet[1] != 0x00) {
      return;
    }

    // In the case of good data, parse the coordinates:
    //
    //   byte 2 = MSB(x)
    //   byte 3 = LSB(x)
    //   byte 4 = MSB(y)
    //   byte 5 = LSB(y)
    x = word(packet[2], packet[3]);
    y = word(packet[4], packet[5]);

    // Convert them from 0..65535 to 0..10000.
    x = (x / 65535.0) * 10000.0;
    y = 10000 - (y / 65535.0) * 10000.0;

    // Move finger to the calculated coordinates.
    Touch.moveFingerTo(FINGER, (int16_t) x, (int16_t) y);

    // We are moving until the interface stops sending data for at least 17ms.
    isMoving = true;
  } else if (isMoving) {
    // When 17ms has passed but no new data arrived, we can release the finger.
    Touch.releaseFinger(FINGER);

    // And we are not moving anymore.
    isMoving = false;
  } else {
    // If there are no data and we are not moving, there is nothing to do
    // besides waiting more 17ms for new data.
  }

  delay(17);
}

How are you testing your code? How are you simulating a loss of data?

I see that you have implemented checks for incorrect data and a method of re-synchronising with the incoming data. That method relies on gaps of at least 5ms between data packets, and if there are gaps between the 6 bytes in a packet, those are less than 5ms. If those timings are reliable then I think your code will recover from a buffer overflow or other loss of data.

Your code now also mentions 17ms gaps between packets. Are you certain this is a safe method? In your first post you said

I don't control how often this device sends the data, assume that it might send data constantly and indefinitely.

My suggested methods to re-synchronise, from reply #4, don't rely on such gaps.

We used an analyzer and there is a delay of 17ms between the packets of data and about 4~5ms to transmit them, I'm guessing this can be relied on, right?

About simulations, I'm using another Arduino to generate similar data and sending packets missing a byte here and there mixed with good packets.

I'm not really experienced with any of that, I even tried to implement using different references of the recvWithStartMarker and recvWithStartEndMarkers from the forum but I don't have an end marker and I have two start markers.

I know it's a matter of adjusting but looking at my last implementation, it seems easier to understand and also seems to be working, or am I missing something here?

I have no idea what can be relied on, I have no idea what is sending this data! But if you have done lots of tests with the logic analyser and you always see those timings, then you should have enough confidence to continue with your plan to use timing to maintain synchronisation, and not need my earlier suggestions as well.

I would suggest not having the delay(17) in your code. As long as you are checking .available() > 0 before you try to read the 6 bytes, you won't trigger the 5ms timeout between packets.

Sorry about that, it is a video interface that hooks into my car factory touch screen and it has this serial wire to report where you touched to integrate with additional modules such as GPS, DVD, TV and etc.

I am using this interface to connect an Android Box and the Arduino to implement the touch HID.

About the delay, when you touch the screen and keep your finger on it, or move it around without releasing, the module keeps sending data (coordinates) every 17ms, each packet takes about 3ms and when you release your finger it stops sending data.

So, after 17ms if there is a new packet in the buffer it means you didn't release the finger yet and the cursor should move around and should not perform a click.

Otherwise, if there are no packets after 17ms, you released it and the driver must perform a click.

That's why the algorithm waits for 17ms before checking if there are new bytes in the buffer before deciding what to do next and why the timeout is 5ms.

The packets are sent in 3ms but I used 5ms to deal with any hiccups that I'm not even sure might happen.

In the end, my concerns are:

  1. If I drag n' drop something for more than a couple of seconds (like moving a Google Maps) it will send a high number of packets to Arduino and will overflow the buffer (or maybe the Arduino can digest all of them in time?)
  2. Also I may have invalid data due to the noisy environment of the car but I guess checking for the first two bytes gives me some confidence that I have a good packet

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.