Need help - Reading serial data from a Datalogger

Hi all,

I’m using an Arduino mega2560 to read data from a datalogger at 9600 baud on RX1 and then printing it to a PC over the USB serial connection. The data comes in 14 byte strings so I have written simple code for the arduino to read the data into a 14 byte array which is to have some math done on certain bytes in the array before being sent to the PC.

The problem I am getting is that after 15 loops the Arduino starts dropping data from the receive buffer as though it cant keep up and no matter what I try it keeps dropping data after 15 loops.

Here is my code:

int start14[14];
        
void setup()
{
	Serial.begin(9600);
        Serial1.begin(9600);
}

void loop()
{
	if(Serial1.available() > 13)
	{
          int x = 0;
            for(x=0;x<14;x++) { 
              start14[x] = Serial1.read();
}
Serial.print(start14[0]);
Serial.print('\32');
Serial.print(start14[1]);
Serial.print('\32');
Serial.print(start14[2]);
Serial.print('\32');
Serial.print(start14[3]);
Serial.print('\32');
Serial.print(start14[4]);
Serial.print('\32');
Serial.print(start14[5]);
Serial.print('\32');
Serial.print(start14[6]);
Serial.print('\32');
Serial.print(start14[7]);
Serial.print('\32');
Serial.print(start14[8]);
Serial.print('\32');
Serial.print(start14[9]);
Serial.print('\32');
Serial.print(start14[10]);
Serial.print('\32');
Serial.print(start14[11]);
Serial.print('\32');
Serial.print(start14[12]);
Serial.print('\32');
Serial.print(start14[13]);
Serial.println();
}
}

And this is what I’m getting on the serial monitor:

0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 0 255 128 78 152 255 253 0 1 2 147 200 0
0 152 255 253 0 1 2 147 200 0 0 0 255 128
78 0 0 0 255 128 78 152 255 253 0 1 2 147
200 0 1 2 147 200 0 0 0 255 128 78 152 255
253 255 128 78 152 255 253 0 1 2 147 200 0 0
0 1 2 147 200 0 0 0 255 128 78 152 255 253
0 78 152 255 253 0 1 2 147 200 0 0 0 255
128 200 0 0 0 255 128 78 152 255 253 0 1 2
147 253 0 1 2 147 200 0 0 0 255 128 78 152
255 0 0 255 128 78 152 255 253 0 1 2 147 200
0 0 1 2 147 200 0 0 0 255 128 78 152 255
253 255 128 78 152 255 253 0 1 2 147 200 0 0
0 1 2 147 200 0 0 0 255 128 78 152 255 253
0 128 78 152 255 253 0 1 2 147 200 0 0 0
255 0 0 0 255 128 78 152 255 253 0 1 2 147
200 0 1 2 147 200 0 0 0 255 128 78 152 255
253 0 0 255 128 78 152 255 253 0 1 2 147 200
0 0 1 2 147 200 0 0 0 255 128 78 152 255
253 255 128 78 152 255 253 0 1 2 147 200 0 0

It continues dropping data from here. Can anyone please help with what I am doing wrong?

And this is what I'm getting on the serial monitor:

Not with that code, it isn't. That code won't even compile.

start13[x] = Serial1.read();

Where is start13 defined?

Sorry I messed that one up when I copied the code to the forum, I've corrected that now. Any ideas whats going on?

Any ideas whats going on?

For every 14 characters received, you send out 30. If the serial data keeps coming in while you are sending data out, which takes longer, eventually the receive buffer will fill up. When that happens, data is discarded.

That looks like what is happening to you.

Do you have any control over the source of the data? If you can make it send data only when you are ready to receive it, you can prevent the buffer overflow problem. If not, you must crank up the speed that you are sending data out, so that outgoing data is sent faster than incoming data arrives.

What is the role of arduino in your project after all? You can just take a USB-TTL adapter and write some software on PC to interpret your data. Will arduino be used to respond to the data or something?

PaulS: For every 14 characters received, you send out 30. If the serial data keeps coming in while you are sending data out, which takes longer, eventually the receive buffer will fill up. When that happens, data is discarded.

That looks like what is happening to you.

Do you have any control over the source of the data? If you can make it send data only when you are ready to receive it, you can prevent the buffer overflow problem. If not, you must crank up the speed that you are sending data out, so that outgoing data is sent faster than incoming data arrives.

Thanks PaulS, that makes sense. Unfortunately I cannot change the source of the data as it is someone elses product and it just continuously transmits data at a constant rate. I'll have to rethink how I tackle this project.

liudr: What is the role of arduino in your project after all? You can just take a USB-TTL adapter and write some software on PC to interpret your data. Will arduino be used to respond to the data or something?

Yes the Arduino is going to perform calculations on certain bytes in the array and then transmit a re-constructed string in a nmea string type format. This is so another piece of software (which I also did not make) on my PC can read this string from a serial port and log/graph the data. Unfortunately I cannot change the source or the destination of the data. The Arduino will hopefully be acting as a translator between strings of data converting one string into another.

What do you mean you can't change the source and destination of the data? Say the program on PC has to read from com 1 or something?

liudr:
What do you mean you can’t change the source and destination of the data? Say the program on PC has to read from com 1 or something?

No the program on the PC can use any com port, but i cannot alter the program in any way. My project is to use the arduino to take data from a datalogger and transform the data into a string that will be read by this piece of software from a com port. I wrote an application in vb6 that does exactly this using a virtual com port emulator to get the vb6 application to talk to the program, and this works fine. But I want to do this with an Arduino (or similar piece of hardware) instead of the vb6 app and was hoping the Arduino would be up to the task.

The following is the full code I had written for the Arduino to communicate between datalogger and software before I realized it was dropping data. I then wittled the code down (which I posted earlier) to find out why it was dropping data.

int i = 0;
int x = 0;
int readbyte[15];
byte readbyte12;
byte readbyte13;
String initString;
String constructString;
char rcString[50];
byte checksum;
        
void setup()
{
	Serial.begin(9600);
        Serial1.begin(9600);
}

void loop()
{
begining:
if(Serial1.available() > 0) {
  readbyte12 = Serial1.read();
  if(readbyte12 == 255) {
    if(Serial1.available() > 0) {
      readbyte13 = Serial1.read();
        if(readbyte13 == 254) {
          Serial.print("check1");
          goto startpoint;
        }
    }
  }
  else if(readbyte12 == 253) {
    if(Serial1.available() > 0) {
      readbyte13 = Serial1.read();
        if(readbyte13 == 254) {
          Serial.print("check2");
          goto startpoint;
        }
    }
  }
  else {
goto begining;
  }
startpoint:   
// check enough bytes are available
	if(Serial1.available() > 13)
	{
          // read bytes to the array
            for(x=0;x<14;x++) { 
              readbyte[x] = Serial1.read();
}
// check the bytes are in the correct locations in the array
if(readbyte[0] != 0){
  goto begining;
}
if(readbyte[1] != 1) {
  goto begining;
}
if(readbyte[2] != 2) {
  goto begining;
}
if(readbyte[12] < 253) {
  goto begining;
}
else if(readbyte[12] == 254) {
  goto begining;
}
if(readbyte[13] != 253) {
  goto begining;
}

 // setup count integer:
  i = i++;
  if (i >= 65535){
      i = 0;}
// Convert bytes to actual usuable values:
  int AFR = readbyte[3];
  int EGT = readbyte[5] * 256 + readbyte[4];
  int RPMDEC = readbyte[7] * 256 + readbyte[6];
  int RPM;
  if (RPMDEC != 0){
  RPM = pow(RPMDEC,-1) * 1152000;
  }
  else { RPM = 0;
  }
  int PON;
  PON = readbyte[9];
  int MAP;
  if (PON > 127){
    MAP = -1;
  }
  else {
    MAP = 1;
  }
  int BOOST = readbyte[8] * MAP;
  int TPS = readbyte[10];
  int USERDEC = readbyte[11];
  int USER;
  if (USERDEC != 0){
     float USERFLOAT = 255.0 / USERDEC;
     USERFLOAT = 5.0 / USERFLOAT;
     USER = USERFLOAT * 100.0;
  }
  else {
    USER = 0;
  }
  
  // create string for Checksum calculation
  initString = "$RC1,,";
  constructString = String(initString + i + ",,,,,,," + RPM + "," + TPS + "," + BOOST + "," + AFR + "," + EGT + "," + USER + ",*");
  constructString.toCharArray(rcString, 50);
  
  // Call Checksum from string10:
  checksum = getCheckSum(rcString);
  
  // send complete string with checksum to serial port
  Serial.print(constructString);
  Serial.println(checksum, HEX);
}
goto startpoint;
}
}

// checksum function
int getCheckSum(char *string) {
 int i;
 int XOR;
 int c;
 // Calculate checksum ignoring any 

s in the string
for (XOR = 0, i = 0; i < strlen(string); i++) {
c = (char)string[i];
if (c == ‘*’) break;
if (c != ’


) XOR ^= c;
 }
 return XOR;
 }

it is someone elses product and it just continuously transmits data at a constant rate

Have you read the manual for the data logger? It is unusual not to be able to configure serial interfaces to use either software or hardware flow control to avoid data loss. Changing the baudrate may also have an effect, you may actually be better with a higher baudrate.

Can you control the baud rate that the program on the PC uses? The fact that you can configure which port it uses gives some hope that you can. If so, per PaulS' earlier comment, just crank up the baud rate that that program uses and adjust the arduino transmission to match.

If not, and you can't control the flow from the datalogger, you're going to need a little external hardware to ignore its transmissions until you have pushed out the data you have.

The best approach is to design the interfaces so that the output has enough bandwidth to handle all the input.

Do you know what rate the input is arriving at? If it's saturating the 9600 baud then you're obviously facing a fundamental problem since you don't have enough bandwidth to send that out.

Increasing the outbound bandwidth is the obvious approach.

If that approach isn't possible, your other option is to discard some of your input stream. As it stands, you will be discarding arbitrary bytes as the serial driver buffers overflow. You don't seem to have any mechanism for getting back in sync with your source. You really should have some way to do that anyway, since serial links can have errors just like any other link. Do you have a byte sequence that you can recognise as the start or end of a frame?

I'd suggest reading bytes from your input stream into memory as they're available, rather than buffering them in the serial driver. This gives you control over which bits to discard if you get into an overflow. One possible approach is to have a pair of swinging buffers - one containing the frame being processed, and one holding the next incoming frame. If your input buffer is full and the output buffer isn't clear yet, you can overwrite the input buffer with the next frame. You lose data, but the overall data path stays intact and everything stays in sync.