Using data from a serial packet

[billvv]

I want to use a Mega to receive serial packets, extract one of the data bytes, and use the value as an input to a PID section that will output a 0-5v control signal to a mosfet driver. For now, I want to test the serial part by connecting it to my known serial stream and display the extracted value as it changes. The specs of the serial stream are:

The interface is async serial at 57.6 kb, 8N1.
Each packet starts with two bytes: 0x55 & 0xAA
There are 273 data bytes following.
Then a two byte checksum. Add together the two beginning of record (BORI) bytes plus the 273 data bytes.
That value (a two byte value) should match the packet check sum to indicate a good packet.
Numbering the data bytes starting at 0 (0 offset into the data area):
Coolant temperature is offset 0xE3, conversion is °C = N * .75 – 40

Is the PacketSerial library the best way to approach decoding and extracting the value?

Here is some C++ code that is being used successfully to parse that serial stream, so it makes sense to me to model it for the arduino...

 //  Packet Parser

 unsigned short int checksum = 0;
 int ProcessEblData(BYTE charval)
 {
 	int pbyte = 0;
 	unsigned short int checkval;
 
 	if ((charval == 0x55) && (packetindex == -1))
 	{
 		// this looks like a start of packet, reset our packet position counter
 		packetindex = 0;
 		// reset our checksum
 		checksum = 0;
 	}
 
 	if ((charval == 0xaa) && (packetindex == 1))
 	{
 
 	}
 	else if ((charval != 0xaa) && (packetindex == 1))
 	{
 
 		
 		packetindex = -1;
 	}
 
 	if (packetindex >= 0)
 	{
 		ebl_packet[packetindex] = charval;
 	        checksum += charval;

 		++packetindex;
 
 	}
 
 	if (packetindex == PACKET_SIZE)
 	{
	
 
 		// reset our index control...
 		packetindex = -1;
 
 		checkval = ebl_packet[PACKET_SIZE - 2] * 0x100;
 		checkval += ebl_packet[PACKET_SIZE - 1];
 
 		// remove the checksum value from the calculated checksum...
 		checksum -= ebl_packet[PACKET_SIZE - 1];
		checksum -= ebl_packet[PACKET_SIZE - 2];
 
 		if (checksum == checkval)
 		{

			++GoodPacketsReceived;
 			ProcessEblPacket();
 
 		}
 		else
 		{
 
 			
 			++BadPacketsReceived;
		}
 
 	}
 
 	return TRUE;
 }
 
 // this is defined as 1 (even though it's two) since the document that 
 // i was referring to used '1' as byte 0.
 #define EBL_DATA_OFFSET 2
 void ProcessEblPacket(void)
 {
 	if (ebl_packet[0x00 + EBL_DATA_OFFSET] == 0xE0)
 	{
 		return;
 	}

     coolant_temp = ebl_packet[0xE3 + EBL_DATA_OFFSET];
     coolant_temp= ((float)coolant_temp * 0.75f) - 40;
     coolant_temp = (coolant_temp * 1.8f) + 32; // convert to degF

[GolamMostafa]

If you understand the sketch that you have posted, then go ahead with that one; else, develop your own serial telemetry.

As the first step of developing your own program, try to simulate the case with 10 bytes wide frame data (2+6+2) between MEGA and UNO.

How has the checksum been computed at the source? Is it normal checksum or the CRC? Why are there two bytes. Normal CHKSUM is usually 1-byte!

[billvv]

Golam,
I didn't realize I could use C++ code in the Arduino IDE, but after your comment did a little googling and realized that's what you meant - I think.

Thanks for the suggestion to model it.

[johnwasser]

@billvv

 	if ((charval == 0x55) && (packetindex == -1))
 	{
 		// this looks like a start of packet, reset our packet position counter
 		packetindex = 0;
 		// reset our checksum
 		checksum = 0;
 	}
 
 	if ((charval == 0xaa) && (packetindex == 1))
 	{
 
 	}

I don't think this is correct. When the 0x55 arrives, set packetindex to 0 but look for packetindex to be 1 when 0xAA arrives.

I would start with packetindex = -2 and set it to -1 when 0x55 arrives. When 0xAA arrives with packetindex == -1 then set packetindex to 0 for the first byte of data.

{
  if (packetindex == -2)
  {
    if (charval == 0x55)
      packetindex++; // First header byte received
    return;
  }

  if (packetindex == -1)
  {
    if (charval == 0xAA)
      packetindex++; // Second header byte received
    else
      packetindex = -2; // Start over
    return;
  }

  if (packetindex < 273)
  {
    ebl_packet[packetindex++] = charval;
    checksum += charval;
    return;
  }

  if (packetindex == 273)
  {
    checkval = charval;
    checkval <<= 8;
    packetindex++;
    return;
  }

  if (packetindex == 274)
  {
    checkval += charval;
    if (checksum != checkval)
    {
      ++BadPacketsReceived;
      packetindex = -2; // Start over
      return;
    }

    // The packet is complete and checked
  }

[GolamMostafa]

@billvv

Thanks for the suggestion to model it.

You may practice the following sketches to receive 10-byte wide binary coded frame (0x55, 0xAA, 0x12, 0x34, 0x56, 0x78, 0x9A, 0xBC, 0x02, 0x6A) by MEGA from UNO (Fig-1 of post #2. (55AA is sync pattern; 026A (last two bytes) is checksum computed by adding all the six data bytes.)

MEGA Sketch: (the receiver)

//receiveing: 55AA 12 34 56 78 9A BC 026A
bool flag1 = false, flag2 = false;
byte myData[8];
int chksum = 0;
int i = 0;

void setup()
{
  Serial.begin(9600);
  Serial1.begin(9600);
}

void loop()
{
  byte n = Serial1.available();
  if (n != 0)
  {
    if (flag2 == false)    //55AA sync pattern is not yet received
    {
      if (flag1 == false)  //55 (part of 55AA) of sync pattern is not yet received
      {
        byte y1 = Serial1.read();
        if ( y1 == 0x55)
        {
          flag1 = true;    //55 is found
          //Serial.print(y1, HEX);  //for debugg
        }
      }
      else
      {
        byte y2 = Serial1.read();
        if ( y2 == 0xAA)
        {
          flag2 = true;
          //Serial.println(y2, HEX);   //for debugg
          Serial.println("Sync pattern is found.");
        }
      }
    }
    else
    {
      myData[i] = Serial1.read();
      if (myData[i] < 0x10)
      {
        Serial.print('0');  //showing leading 0 on Serial Monitor
      }
      Serial.print(myData[i], HEX); Serial.print(' ');
      i++;
      if (i == 8)
      {
        i = 0;
        flag1 = false;
        flag2 = false;
        Serial.println();
        for(int j = 0; j<6; j++)  //compute checksum from received frame excluding sync
        {
          chksum += myData[j];
        }
        Serial.println(chksum, HEX);
        if(chksum == (myData[6]<<8|myData[7]))
        {
          chksum = 0;
          Serial.println("Valid frame is received.");
        }
      }
    }
  }
}

UNO Sketch: (the simulator)

#include<SoftwareSerial.h>
SoftwareSerial SUART(2, 3);  //DPin=2 = SRX, DPin-3 = STX
byte myData[] =  {0x55, 0xAA, 0x12, 0x34, 0x56,    //simulated frame
                  0x78, 0x9A, 0xBC, 0x02, 0x6A
                 };
int i = 0;

void setup()
{
  Serial.begin(9600);
  SUART.begin(9600);
}

void loop()
{
  SUART.write(myData[i]);
  if(myData[i] < 0x10)
  {
    Serial.print('0');
  }
  Serial.print(myData[i], HEX); Serial.print(' ');
  i++;
  if (i == 10)
  {
    i = 0;
    Serial.println();
  }
  delay(1000); //test interval
}

MEGA's Monitor:

55AA
Sync pattern is found.
12 34 56 78 9A BC 02 6A 
26A
Valid frame is received.

[billvv]

Thanks for the modelling logic! I will try that and perhaps try to adapt it for my situation. I've done some more work based on the C++ code for a driver for the WinLog app, and think I've got that form of logic correct now; at least the packet grab and parsing... I have a uno and a mega, and a string that has a known good packet so I think I have all that I need. Thanks especially for the simulator code; I will be able to use that for testing this and the next step, which is PID control with PWM output to control an electric cooling fan. (for an old motorhome). Probably just 'P' control.

{0x55, 0xAA, 0x00, 0x02, 0x80, ....a bunch more bytes... 0x02, 0x35, 0x01, 0x55, 0xAA} (the checksum is correct per my spreadsheet).

// adapted from Serial Input Basics
//https://forum.arduino.cc/t/serial-input-basics/278284/70
// char data type replaced by byte
// From Dynamic_EFI.....
//The interface is async serial at 57.6 kb, 8N1.
//Each packet starts with two bytes: 0x55 & 0xAA
//There are 273 data bytes following.
//Then a two byte checksum. Add together the two beginning of record (BORI) bytes plus the 273 data bytes.
//That value (a two byte value) should match the packet check sum to indicate a good packet.
//Numbering the data bytes starting at 0 (0 offset into the data area):
//Coolant temperature is offset 0xE3, conversion is °C = N * .75 – 40
//.... End Dynamic_EFI


const byte numBytes = 277; //BVV
#define EBL_DATA_OFFSET 2
byte receivedBytes[numBytes];
byte numReceived = 0;
unsigned short int checksum = 0; 
unsigned short int checkval;
unsigned long coolant_temp = 0;
unsigned char ebl_packet[numBytes];
boolean newData = false;

void setup() {
    //Serial.begin(9600);
    Serial.begin(57600); //BVV  Mega 
    Serial.println("<Arduino is ready>");
    Serial1.begin(9600);
}

void loop() {
  recvWithStartLengthAndChecksum(); // BVV
  ProcessEblPacket();
    showNewData();
}

void recvWithStartLengthAndChecksum() {
    static boolean recvInProgress = false;
    static byte ndx = -1;
    byte startMarker0 = 0x55; //BVV changed for EBL
    byte startMarker1 = 0xaa;
    byte rb;

  while (Serial.available() > 0 && newData == false) {
    rb = Serial.read();

     if ((rb == startMarker0) && (ndx == -1)){ //start of packet (SM0)
      ndx=0; //reset packet position counter (index)
      checksum=0; //reset checksum
    }

    if ((rb == startMarker1) && (ndx == 1)) {  //second byte (SM1)of packet
    } else if ((rb !=startMarker1) && (ndx == 1)) {  
      ndx = -1; //wrong second byte so keep looking for SM0
    }
            
    if (ndx>=0) {  //good byte
      ebl_packet[ndx] = rb;  //add byte to packet
      checksum += rb;  // add byte value to checksum
      ++ndx; //increment index
    }
    
    if (ndx == numBytes)  {  //if full 277 byte packet 
      ndx= -1;             //reset index for next packet
      // derive checksum value from last two bytes
      checkval = ebl_packet[numBytes -2] * 0x100;
      checkval += ebl_packet[numBytes -1]; 
      // subtract the last two bytes from the summed checksum.
      checksum -= ebl_packet[numBytes -1];
      checksum -= ebl_packet[numBytes -2];
      
      if (checksum == checkval); {  //valid packet
        numReceived = ndx;  // save the number for use when printing
        newData = true;
      }   
    }   //END if full 277 byte packet
  } //END while serial data available
}   // END recvWithStartLengthAndChecksum



void ProcessEblPacket() {
  coolant_temp = ebl_packet[0xE3 + EBL_DATA_OFFSET];
  coolant_temp= ((float)coolant_temp * 0.75f) - 40;
  coolant_temp = (coolant_temp * 1.8f) + 32; // convert to degF
}

void showNewData() { //Shows full packet
    if (newData == true) {
        Serial.print("This just in ... ");
        for (byte n = 0; n < numReceived; n++) {
            Serial.print(receivedBytes[n], HEX);
            Serial.print(' ');
        }
        Serial.println();
        showGroupsOfBytes();
        newData = false;
    }
}

void showGroupsOfBytes() {
    for (byte n = 0; n < numReceived; n++) {
        Serial.print(receivedBytes[n], HEX);
        Serial.print(' ');
        if ((n + 1) % 5 == 0) {
            Serial.println();
        }
    }
    Serial.println();
}

[GolamMostafa]

Have you tested your sketch of post #6 with the simulated codes of post #5?

[billvv]

Haven't yet; I'm away for a few days but will do when I get back.

I assume the left bitshift and 'or' adds the two values for the checksum? " (myData[6]<<8|myData[7]))"

[nicolajna]

Keep in mind that by default the RX buffer on the AVR core is either 16bytes (if ram is less than 1023) or 64 bytes. Unless increasing the size you should take care not to let it overflow.

See: ArduinoCore-avr/HardwareSerial.h at master · arduino/ArduinoCore-avr · GitHub
Line: 50-54

[GolamMostafa]

@nicolajna

Keep in mind that by default the RX buffer on the AVR core is either 16bytes (if ram is less than 1023) or 64 bytes. Unless increasing the size you should take care not to let it overflow.

The FIFO serial buffer is 64 bytes wide in 328P of the UNO Board. If OP is going to use non-blocking code, then buffer size is not concerned.

[nicolajna]

@GolamMostafa

The FIFO serial buffer is 64 bytes wide in 328P of the UNO Board.

He's using an ATmega2560 (I figure since he says Mega) not an ATmega328. Neither have a 64 byte deep FIFO, it's 2 bytes deep. The serial driver in the Arduino framework however implements a (by default) 16 or 64 byte ring buffer, which can be changed if he so require.

@GolamMostafa

If OP is going to use non-blocking code, then buffer size is not concerned.

I disagree! Buffer sizes are always of concern. In either case I only reminded him to take care.

[noiasca]

@billvv

Is the PacketSerial library the best way to approach decoding and extracting the value?

imho the best start is to follow the tutorial "Serial Input Basic" from this forum.

Serial Input Basics - updated

Updated Version Please note that this is a revised version of the advice in this earlier Thread which has become very long. As far as possible I have kept the code examples identical or simplifed them slightly. It should not be necessary to refer to the older Thread, but feel free to do so. Contents The following sections are in this Tutorial Introduction Serial data is slow by Arduino standards Example 1 - Receiving single characters Why code is organized into functions Exampl…

just two days ago, we have solved another "read message from a device" - it should give you an idea how to solve such reading.

How to parse serial data string to Floating values

this will read such a message from Serial and print the values every 3 seconds to serial /* Serial Receive Example no start tag [Tag Value ValueEndtag] separated values by ValueEndeTag D no dedicated end tag C1.8D1V0.0DWh0.0DAh0.0D where C is current V is voltage wh is watthour ah amperehour this variant parses each value during the incoming packet by noiasca 2022-06-26 https://forum.arduino.cc/t/how-to-parse-serial-data-string-to-integers/1006661 */ …

if you need further support:

• we would need a product link of the device you want to read (otherwise it makes no sense to invest time to an unknown product)
• a datasheet describing the message protocol
• two FULL messages as an example including an indication which values are from interest for you.

[billvv]

Thanks for all the help here... Post 6 specifies the message protocol provided by the manufacturer; I am only wanting to read and save the coolant temperature from that packet.

//The interface is async serial at 57.6 kb, 8N1.
//Each packet starts with two bytes: 0x55 & 0xAA
//There are 273 data bytes following.
//Then a two byte checksum. Add together the two beginning of record (BORI) bytes plus the 273 data bytes.
//That value (a two byte value) should match the packet check sum to indicate a good packet.
//Numbering the data bytes starting at 0 (0 offset into the data area):
//Coolant temperature is offset 0xE3, conversion is °C = N * .75 – 40

I have two programs that will read these packets, one is the manufacturer's and one is a driver created for us by the developer of WinLog. Both programs work perfectly from this proyocol. I was able to set up a uno to transmit a known good packet and the mega to receive and display the packet and that worked just fine. It found the start bytes, stored the next 273 bytes, and proved the checksum. I have the Winlog developers C++ code revised and compiled as an Arduino sketch but have not tested it. I may not, since the Post 5 code below works fine; I modified the receive code in Post 5 for this packet and it works fine as well (at least I see the packets and newlines between them in the Serial Monitor).

#include<SoftwareSerial.h>
SoftwareSerial SUART(2, 3);  //DPin=2 = SRX, DPin-3 = STX
byte myData[] =  {0x55, 0xAA, 0x00, 0x02, 0x80, 0x08, 0x08, 0x24, 0x80, 0x20, 0x40, 0x01, 0x11, 
                  0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00,
                  0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xCC, 0x4C, 0xF9, 0x4D, 
                  0xCC, 0xFF, 0x00, 0x00, 0xF3, 0x00, 0x00, 0xF3, 0xF3, 0xC8, 0xFF, 0x00, 0xFF, 
                  0x00, 0xFF, 0x00, 0x00, 0x04, 0x77, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x64, 
                  0x64, 0x00, 0x67, 0xF8, 0x66, 0x01, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                  0x00, 0x00, 0x00, 0x3F, 0xC0, 0x00, 0x00, 0xF3, 0x00, 0xB6, 0x40, 0xF9, 0x66, 
                  0x00, 0x00, 0x00, 0x00, 0x0A, 0x27, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                  0x00, 0x00, 0x00, 0x00, 0x86, 0xFF, 0xF8, 0x00, 0x00, 0x00, 0x61, 0x00, 0x17, 
                  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x04, 0x00, 0x4D, 0x6B, 0x01, 
                  0xCC, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 
                  0x00, 0x00, 0x00, 0x00, 0x72, 0x00, 0x00, 0x00, 0x00, 0x00, 0x54, 0x5E, 0x00, 
                  0x00, 0x00, 0x00, 0x53, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x80, 0x80, 0x27, 0x00, 
                  0x0A, 0xEC, 0xC5, 0xAC, 0x0A, 0xD1, 0x00, 0x00, 0x00, 0x16, 0x00, 0x80, 0x00, 
                  0x00, 0x00, 0x27, 0xF7, 0x00, 0x00, 0x00, 0x00, 0x00, 0x64, 0x1A, 0xFF, 0x00, 
                  0x20, 0x5C, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x73, 0x08, 0x00, 
                  0x0D, 0x35, 0x46, 0x50, 0x00, 0x64, 0xDE, 0x58, 0xC8, 0xF9, 0x0E, 0xEA, 0xB6, 
                  0xDF, 0xB4, 0x64, 0x00, 0x20, 0x02, 0x00, 0x00, 0x4D, 0x16, 0x19, 0x19, 0x15, 
                  0x19, 0x00, 0x04, 0x00, 0x00, 0x80, 0x80, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 
                  0x00, 0x00, 0xC6, 0x75, 0x2C, 0x05, 0x40, 0x1C, 0x73, 0xF3, 0x00, 0x90, 0x00, 
                  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 
                  0x00, 0x02, 0x35, 0x01
                  };
                  
int i = 0;

void setup()
{
  Serial.begin(57600);
  SUART.begin(57600);
}

void loop()
{
  SUART.write(myData[i]);
  if(myData[i] < 0x10)
  {
    Serial.print('0');
  }
  Serial.print(myData[i], HEX); Serial.print(' ');
  i++;
  if (i == 277)
  {
    i = 0;
    Serial.println();
  }
  delay(100); //test interval
}

Here is the adapted receive code for the Mega that receives packets from the simulator code above for the Uno: For some reason I don't see "Sync pattern is found" or "Valid frame is received" on the serial monitor, even though the packet bytes show and there's a line feed between packets.

//receiveing: 55AA , etc.
bool flag1 = false, flag2 = false;
byte myData[277];
int chksum = 0;
int i = 0;

void setup()
{
  Serial.begin(57600);
  Serial1.begin(57600);
}

void loop()
{
  byte n = Serial1.available();
  if (n != 0)
  {
    if (flag2 == false)    //55AA sync pattern is not yet received
    {
      if (flag1 == false)  //55 (part of 55AA) of sync pattern is not yet received
      {
        byte y1 = Serial1.read();
        if ( y1 == 0x55)
        {
          flag1 = true;    //55 is found
          //Serial.print(y1, HEX);  //for debugg
        }
      }
      else
      {
        byte y2 = Serial1.read();
        if ( y2 == 0xAA)
        {
          flag2 = true;
          //Serial.println(y2, HEX);   //for debugg
          Serial.println("Sync pattern is found.");
        }
      }
    }
    else
    {
      myData[i] = Serial1.read();
      if (myData[i] < 0x10)
      {
        Serial.print('0');  //showing leading 0 on Serial Monitor
      }
      Serial.print(myData[i], HEX); Serial.print(' ');
      i++;
      if (i == 277)
      {
        i = 0;
        flag1 = false;
        flag2 = false;
        Serial.println();
        for(int j = 0; j<275; j++)  //compute checksum from received frame excluding sync
        {
          chksum += myData[j];
        }
        Serial.println(chksum, HEX);
        if(chksum == (myData[276]<<8|myData[277]))
        {
          chksum = 0;
          Serial.println("Valid frame is received.");
        }
      }
    }
  }
}

[GolamMostafa]

@nicolajna

I disagree! Buffer sizes are always of concern. In either case I only reminded him to take care.

In non-blocking sketch, a data/character arrives and the user program reads it; the serial buffer immediately becomes empty. So, there is only one location involved.

[GolamMostafa]

1.

@billvv

  if (i == 277)
  {

Should be:

 if (i == 275)
  {

as index i has started from 0 after passing behind first 2 sync bytes.

2.

@billvv

for(int j = 0; j<275; j++)  //compute checksum from received frame excluding sync
        {
          chksum += myData[j];
        }
        Serial.println(chksum, HEX);
        if(chksum == (myData[276]<<8|myData[277]))
        {

Should be:

 for(int j = 0; j<273; j++)  //compute checksum from received frame excluding sync
        {
          chksum += myData[j];
        }
        Serial.println(chksum, HEX);
        if(chksum == (myData[273]<<8|myData[274]))

3.
Be sure that you get the following two messages on the Serial Monitor of MEGA; otherwise, the received frame is not valid.

"Sync pattern is found."
"Valid frame is received."

[nicolajna]

@GolamMostafa

In non-blocking sketch, a data/character arrives and the user program reads it;

I'm familiar with how asynchronous programming works. And I agree that the poster should write non-blocking code.

@nicolajna

Unless increasing the size you should take care not to let it overflow.

Which is why I posted to take care.

@GolamMostafa

So, there is only one location involved.

I have no idea what this means.

[billvv]

Thanks, Golam...
I didn't think that 277 vs 275 through very well.
I think I need to look at the checksum logic... according to the protocol description, checksum is the sum of the first two bytes plus the next 273 data bytes, so I'd have to add the first two to checksum and then sum the next 273.

You have been a great help; thank you.

[GolamMostafa]

@billvv

the sum of the first two bytes plus the next 273 data bytes, so I'd have to add the first two to checksum and then sum the next 273.

In checksum calculation, the sync bytes are not usually included.

The 277-byte wide array of your post #13 shows checksum = 0x34, 0x02; whereas, you have shown 0x35, 0x01.

Serial.begin(9600);
  int chksum = 0;
  Serial.println(sizeof myData);  //277
  for (int i = 2; i < 275; i++)  //277 - 2 -2 = 273 byte information 
  {
    chksum += myData[i];
  }
  Serial.print(chksum, HEX);   //shows: 3402

[billvv]

Sorry for the delay in responding...
Thanks to the help from Golam, plus the known good 'C' code, I can now receive my packets (Uno simulator and Mega receive) and extract the value I'm looking for!

And now I have a question on structure..

Once I have the value, I need to create PID code to derive a control signal, and finally create a PWM value to send to the controller. My question is... Should the Packet, PID, and PWM code be in functions, or just parts of a long 'void loop()' section?