Need help reading serial data

Vaikis · December 5, 2019, 9:52am

Hi, I'am doing some serial decode - protocol lenght is 202. I know begining and end.
Begining is always data[0] = 71, data[1] = C8, last data[202] = 8-bit checksum

8-bit Checksum to verify serial data ( sum(all bytes) & 0xFF == 0 )

Question what is best way do define that readbytes would be from start 71 + C8 till end or till next 71+C8 ? THANKS !

Now my readserial looks like:

void readSerial()

{

if (Serial.available() > 0) {

data_length = Serial.readBytes(data, 202);

Serial1.println(String(data));

while (Serial.available()) {

delay(2);

Serial.read();

}

else data_length = 0;

}

Robin2 · December 5, 2019, 11:04am

Always post a complete program.

It is not a good idea to use the String (capital S) class on an Arduino as it can cause memory corruption in the small memory on an Arduino. This can happen after the program has been running perfectly for some time. Just use cstrings - char arrays terminated with '\0' (NULL).

I think your proposed system will suffer because it is not identifying the start of the message which could mean that it counts 202 bytes from the wrong start point.

Is there any means to identify the end of the message? For example is there a measurable time between the end of one message and the start of another?

If there is I think I would check the time occasion I read from the serial input buffer and if the interval is long enough I would conclude that I had the end of the message. Something like this pseudo code

if (Serial.available > 0) {
   mostRecentDataTime = millis();
   while (Serial.available > 0) {
       receivedChars[nextPos) = Serial.read();
       nextPos ++;
       if (nextPos >= maxChars) {
          nextPos = maxChars - 1;
       }
    }
}
if (millis() - mostRecebtDataTime >= longInterval) {
   // we now have all the data in this message
}

After you know you have the end of the message you can then examine what you have to see if it is valid

I'm assuming you have a char array called receivedChars[] which is big enough to hold your entire message.

Have a look at the second example in Serial Input Basics from which these ideas have been evolved. The example should give you all the background you need.

...R

Vaikis · December 5, 2019, 11:13am

#define MAXDATASIZE 256

char data[MAXDATASIZE];

int data_length = 0;

End message will be too hard to catch using just time.... End mesage is a 8-bit Checksum [202] value.

Vaikis · December 5, 2019, 12:04pm

good data will be
starting from:
byte startMarker0 = 0x71;
byte startMarker1 = 0xc8;
till the end with size of 202

here is answer which required

71 c8 01 10 56 55 62 49 00 05 00 00 00 00 00 00 00 00 00 00 19 15 11 55 16 5e 55 05 09 00 00 00 00 00 00 00 00 00 80 8f 80 8a b2 71 71 97 99 00 00 00 00 00 00 00 00 00 00 00 80 85 15 8a 85 85 d0 7b 78 1f 7e 1f 1f 79 79 8d 8d 9e 96 71 8f b7 a3 7b 8f 8e 85 80 8f 8a 94 9e 8a 8a 94 9e 82 90 8b 05 65 78 c1 0b 00 00 00 00 00 00 00 00 55 56 55 21 53 15 5a 05 12 12 19 00 00 00 00 00 00 00 00 e2 ce 0d 71 81 72 ce 0c 92 81 b0 00 aa 7c ab b0 32 32 9c b6 32 32 32 80 b7 af cd 9a ac 79 80 77 80 ff 91 01 29 59 00 00 3b 0b 1c 51 59 01 36 79 01 01 c3 02 00 dd 02 00 05 00 00 01 00 00 06 01 01 01 01 01 0a 14 00 00 00 77

ToddL1962 · December 5, 2019, 1:17pm

The protocol defines the first 2 bytes in the message are 0x71c8 and assumes that 0x71c8 will never appear in the data, if I understand correctly.

Here is how I would approach it:

read serial data until 0x71c8 is encountered.
read until either 200 bytes are read or until a specified timeout
if timeout, then start over at 1
compute checksum and compare
if no errors process message
back to 1 to read next message

anon26818118 · December 5, 2019, 3:27pm

can try this method

read 1 char. if == startchart storeit ELSE discard it, continue on 1
read 1 more char. if == 2nd char storeit ELSE discard ALL, continue on 1
read 200 more char and store them

verify
Is 1st char ok?
Is 2nd char ok?
Calculate checksum of received. is it == endchar?

Robin2 · December 5, 2019, 4:27pm

Vaikis:
End message will be too hard to catch using just time....

Why?

What is the interval between messages?

As long as the gap between messages is some multiple of the interval between characters it should be trivial to detect the end.

...R

Vaikis · December 5, 2019, 4:40pm

ToddL1962:
The protocol defines the first 2 bytes in the message are 0x71c8 and assumes that 0x71c8 will never appear in the data, if I understand correctly.

Here is how I would approach it:

read serial data until 0x71c8 is encountered.

read until either 200 bytes are read or until a specified timeout

if timeout, then start over at 1

compute checksum and compare

if no errors process message

back to 1 to read next message

how it could look in code ?

ToddL1962 · December 5, 2019, 7:10pm

Vaikis:
how it could look in code ?

You should attempt the code yourself then we can help you through it.

Vaikis · December 6, 2019, 6:07am

ToddL1962:
You should attempt the code yourself then we can help you through it.

but how to check if second value 0xc8 is after the first one 0x71 ?

bool new_data = false;
char receivedChars[buffsize];   

void RecvWithStartMark() {

   static byte ndx = 0;
   byte startMarker = 0x71;
   byte rc;
   

   while (Serial.available() > 0 && new_data == false) {
       rc = Serial.read();
       if (rc != startMarker) {
           receivedChars[ndx] = rc;
           ndx++;
           if (ndx >= buffsize) {
               ndx = buffsize - 1;
           }
       }
       else {
           receivedChars[ndx] = '\0'; // terminate the string
           ndx = 0;
           new_data = true;
       }
       yield();
   }
}

GolamMostafa · December 6, 2019, 6:21am

In your Post#3, there are 200 bytes data items excluding the fixed preamble (0x71C8) and the last variable checksum (0x77 for this frame). You want to catch them and store in an array -- correct?

In checksum calculation, is the preamble code (0x71C8) included? I mean if the checksum is computed on 2 + 200 = 202 bytes of data items?

Which Arduino (UNO, NANO, MEGA, DUE, Node) you are using?

Vaikis · December 6, 2019, 6:55am

GolamMostafa:
In your Post#3, there are 200 bytes data items excluding the fixed preamble (0x71C8) and the last variable checksum (0x77 for this frame). You want to catch them and store in an array -- correct?

In checksum calculation, is the preamble code (0x71C8) included? I mean if the checksum is computed on 2 + 200 = 202 bytes of data items?

Which Arduino (UNO, NANO, MEGA, DUE, Node) you are using?

I'am using ESP8266 based board Wemos D1 Mini. Checksum includes all 201 bytes - 202 is a checksum value. I need to store and read all 202 bytes.
there is no 0x71C8, its two seperate values 0x71 and 0xc8

Robin2 · December 6, 2019, 9:40am

Vaikis:
there is no 0x71C8, its two seperate values 0x71 and 0xc8

You should ignore all values until you receive an 0x71. Then if the next byte is 0xC8 you know you have the header. if it is not 0xC8 go back to searching for 0x71.

Things are a little more complicated if the second byte also happens to be 0x71 as, in that case, you should continue checking for a subsequent 0xC8.

Think of the logic as a series of states

waiting-for-0x71
waiting-for-0xC8
receiving-message

Note that there is no need to save the 0x71 and 0xC8 to the received array as they arrive. You can always put them in afterwards. Or have them permanently in the array and just start saving data from the 3rd character position onward.

...R

Vaikis · December 6, 2019, 11:44am

Iam totaly "green" in cooding..... so it will take ages for me to figure out and I need to decode protocol....

in this example I understand that only when will be begining 0x71 we get all bytes, how coreclty to add second condition, another if condition after the first ?

const byte numBytes = 202;
byte receivedBytes[numBytes];
byte numReceived = 0;

boolean newData = false;


void loop() {
    recvBytesWithStartEndMarkers();
    showNewData();
}

void recvBytesWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    byte startMarker0 = 0x71;
    byte startMarker1 = 0xc8;
    byte rb;
    

    while (Serial.available() > 0 && newData == false) {
        rb = Serial.read();

        if (recvInProgress == true) {
            if (rb != startMarker0) {
                receivedBytes[ndx] = rb;
                ndx++;
                if (ndx >= numBytes) {
                    ndx = numBytes - 1;
                }
            }
            else {
                receivedBytes[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                numReceived = ndx;  // save the number for use when printing
                ndx = 0;
                newData = true;
            }
        }

        else if (rb == startMarker) {
            recvInProgress = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        for (byte n = 0; n < numReceived; n++) {
            Serial.print(receivedBytes[n], HEX);
            Serial.print(' ');
        }
        Serial.println();
        showGroupsOfBytes();
        newData = false;
    }
}

void showGroupsOfBytes() {
    for (byte n = 0; n < numReceived; n++) {
        Serial.print(receivedBytes[n], HEX);
        Serial.print(' ');
        if ((n + 1) % 5 == 0) {
            Serial.println();
        }
    }
    Serial.println();
}

Robin2 · December 6, 2019, 1:39pm

Try this in place of your recvBytesWithStartEndMarkers(). I'm not sure why you started with that example as you have no end marker

void recvWithStartAndLength() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    byte startMarker0 = 0x71;
    byte startMarker1 = 0xc8;
    byte rb;
    

    while (Serial.available() > 0 && newData == false) {
        rb = Serial.read();

        if (rb == startMarker0) {
            waitingForStartMarker1 = true;
            continue; // we go on to read the next byte
        }

        if (waitingForStartMarker1 == true) {
            if (rb == startMarker1) {
                waitingForBody = true;
                ndx = 2; // save the body from the 3rd byte onwards
                        // first 2 bytes are always 0x71 and 0xC8
                continue;
            }
            else {
               waitingForStartMarker1 = false;
            }
        }
        
        if (waitingForBody == true) {
            receivedBytes[ndx] = rb;
            ndx++;
            if (ndx >= numBytes) {
                ndx = numBytes - 1;
                     // your project may not need this next line
                // receivedBytes[ndx] = '\0'; // terminate the string
                waitingForBody = false;
                numReceived = ndx;  // save the number for use when printing
                newData = true;
            }
        }
}

...R

GolamMostafa · December 6, 2019, 6:03pm

Example sketches tested on ESP8266-NodeMCU (Receiver) and NANO(Sender).
ESP8266 (Receiver) Codes:

#include<SoftwareSerial.h> //ESP8266 NodeMCU
SoftwareSerial SUART(D5, D6);  //SRX = D5, STX = D6
bool flag = false;
byte i = 0;
byte m;
byte x1;
byte z1, z2;
byte CHKSUM = 0;

byte myData[201];

void setup()
{
  Serial.begin(115200);
  SUART.begin(9600);
  Serial.println();
}

void loop()
{
  byte n = SUART.available();
  if (n != 0)
  {
    x1 = SUART.read();
    if (flag == false)
    {
      if (x1 == 0x71)
      {
        z1 = x1;
        flag = true;
      }
    }
    else
    {
      if (x1 == 0xC8)
      {
        z2 = x1;
        Serial.println("===================");
        Serial.println("Preamble Found.");
        m = SUART.readBytes(myData, 201);
        //  Serial.println(m, DEC);  //debug
        Serial.print("Received CHKSUM: "); Serial.println(myData[200], HEX); //debug
        for (byte i = 0; i < 200; i++)
        {
          CHKSUM = CHKSUM + myData[i];
        }
        CHKSUM = CHKSUM + z1 + z2; //z1 = 0x71, z2 = 0xC8
        CHKSUM = ~CHKSUM;
        CHKSUM = CHKSUM + 1;
        Serial.print("Computed CHKSUM: "); Serial.println(CHKSUM, HEX);//debug
        if (CHKSUM == myData[200])
        {
          Serial.println("Received Valid Data.");
          CHKSUM = 0;
        }
      }
    }
  }
}

NANO(Sender) Codes:

#include<SoftwareSerial.h>  //NANO
SoftwareSerial SUART(2, 3);//SRX = D2(2), STX = D6
int CHKSUM = 0;
byte CHKSUMT;

byte myData[] =
{
  0x71, 0xc8, 0x01, 0x10, 0x56, 0x55, 0x62, 0x49, 0x00, 0x05,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x19, 0x15, 0x11, 0x55, 0x16, 0x5e, 0x55, 0x05, 0x09, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x8f,
  0x80, 0x8a, 0xb2, 0x71, 0x71, 0x97, 0x99, 0x00, 0x00, 0x00,
  0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x80, 0x85,
  0x15, 0x8a, 0x85, 0x85, 0xd0, 0x7b, 0x78, 0x1f, 0x7e, 0x1f,
  0x1f, 0x79, 0x79, 0x8d, 0x8d, 0x9e, 0x96, 0x71, 0x8f, 0xb7,
  0xa3, 0x7b, 0x8f, 0x8e, 0x85, 0x80, 0x8f, 0x8a, 0x94, 0x9e,
  0x8a, 0x8a, 0x94, 0x9e, 0x82, 0x90, 0x8b, 0x05, 0x65, 0x78,
  0xc1, 0x0b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
  0x55, 0x56, 0x55, 0x21, 0x53, 0x15, 0x5a, 0x05, 0x12, 0x12,
  0x19, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xe2,
  0xce, 0x0d, 0x71, 0x81, 0x72, 0xce, 0x0c, 0x92, 0x81, 0xb0,
  0x00, 0xaa, 0x7c, 0xab, 0xb0, 0x32, 0x32, 0x9c, 0xb6, 0x32,
  0x32, 0x32, 0x80, 0xb7, 0xaf, 0xcd, 0x9a, 0xac, 0x79, 0x80,
  0x77, 0x80, 0xff, 0x91, 0x01, 0x29, 0x59, 0x00, 0x00, 0x3b,
  0x0b, 0x1c, 0x51, 0x59, 0x01, 0x36, 0x79, 0x01, 0x01, 0xc3,
  0x02, 0x00, 0xdd, 0x02, 0x00, 0x05, 0x00, 0x00, 0x01, 0x00,
  0x00, 0x06, 0x01, 0x01, 0x01, 0x01, 0x01, 0x0a, 0x14, 0x00,
  0x00, 0x00
};

void setup()
{
  Serial.begin(115200);
  SUART.begin(9600);
  for (byte i = 0; i < sizeof myData; i++)
  {
    CHKSUM += myData[i];
  }
  CHKSUMT = lowByte(CHKSUM);
  CHKSUMT = ~CHKSUMT;
  CHKSUMT = CHKSUMT + 1;
  // SUART.write(CHKSUMT);
  Serial.print(CHKSUMT, HEX);
 // myData[201] = CHKSUMT;

}

void loop()
{
  SUART.write(myData, sizeof myData);
  SUART.write(CHKSUMT);
  Serial.println(CHKSUMT, HEX);
  delay(1000);
}

Receiver Screenshot:

Vaikis · December 9, 2019, 7:39am

thanks guys for help !!!!

Topic		Replies	Views
Sending byte array over Serial comm Programming	31	980	October 28, 2024
[Solved] Read Serial Byte in HEX Networking, Protocols, and Devices	47	23079	May 6, 2021
Reading serial data packets General Guidance	30	278	July 24, 2025
Serial communication isuue General Guidance	10	112	August 25, 2025
Is there a way to read parts of incoming serial? Programming	26	4258	May 5, 2021

Need help reading serial data

Related topics