Access infinity long hex data receiving from UART rx

hello everyone,

I want to ask how to access long hex data from the UART RX pin. I have a device that sends me data through the UART protocol non-stop. There are many addresses to capture different types of data and the starting hex string needs to be identified in order to get the data. In simple language,

think there is a data hex string as follows. (example)

0xA896565.........55AA67678734.......8989

This hex data is received endlessly. The 55AA is the starting bits (Packet header). then next 67 represent data of the angle. after the packet header and 64bits, [5566+64bits]then the next 128 bits represent the distance. how can identify the 55AA starting header packet and access the next bits respectively?

image
there are many bit streams like this above data sharing way. ex power indication bit streams, product details, and versions bit streams are there too. So it's like endless data sharing.

my problem is to access starting bits and respective bits mentioned. a sample code is much appreciated and thank you.

And how would you do it yourself in the place of the program? - read incoming bytes, if caught 55 - check the next one and if it is AA - the problem is solved

What is the baudrate?
What you need to do with the received data?

Read the incoming data until you get an 0x55
Read the next byte and if it is not 0xAA then go back to reading until you get an 0x55
Keep going until you read 0x55 and 0xAA in successive bytes. At that point you will know that the next byte contains the start of the data so read it and do whatever you need with it

do you have any sample codes regarding this? I have tried to identify using a program like this.

#include <SoftwareSerial.h>
SoftwareSerial s(19,18);
int x[64];
int state= 0;
      
       
void setup() 
{      
     Serial.begin(9600);
     Serial1.begin(115200);
}
 
void loop() 
{   
    if(Serial1.available()>0) 
    {
        for(int i=0; i<=64;i++){
         x[i] = Serial1.read();    
              
         
            if(x[i]==0xAA && state ==1){
             Serial.println("found starting bits ");}
              if(x[i]==0x55) {state=1; }else {state =0;}
          }
    }
        delay(1000);  //test interval
}

its not working I guess. baud rate is 115200 device - ydlidar x2
I'm going to calculate the angle and distance, then going to plot a map

Do you have any sample code, sir? I don't receive any header in my code. is the process ok

What is purpose of SoftwareSerial in the code?

it's just defining the 19th pin as my RX.

From which port do you read your "long hex data" - Serial Serial1 or SoftSerial?
What is your board?
What pins do you use as Rx TX for ydlidar x2 ?

serial 1 in pin 19. the data is receiving but start bits couldn't identify. lidar pins tx has set to pin 19 Rx in Arduino mega

You dont need "setting the RX pin" by Softwareserial. It completely nonsense. Remove this two first lines from the code.

And now, you code in the loop is a mess.

If you checked (Serial.available > 0) is true - it means that there is at least one byte in the Serial. So you should read ONE byte at once and check if the next available. Why do you try to read 64 bytes? Think - what will be the result if serial contains only four bytes, for example?

    if(Serial1.available()>0) 
    {
        for(int i=0; i<=64;i++){
         x[i] = Serial1.read();    

Whatever else is wrong this won't work because after testing that at least 1 byte is available you proceed to blindly read 64 bytes not even knowing that you are starting at the right place

To my mind you would be better off using a state machine using 4 states WAITING_FOR_AA, WAITING_FOR_55, READING_DATA and PROCESSING_DATA

Something along these lines

enum states
{
  WAITING_FOR_AA,
  WAITING_FOR_55,
  READING_DATA,
  PROCESSING_DATA
};
byte currentState = WAITING_FOR_AA;
const byte dataLength = 8;
byte data[dataLength];
void setup()
{
  Serial.begin(115200);
}

void loop()
{
  switch (currentState)
  {
    case WAITING_FOR_AA:
      if (Serial.read() == 0xAA)
      {
        currentState = WAITING_FOR_55;
      }
      break;
    case WAITING_FOR_55:
      if (Serial.read() == 0x55)
      {
        currentState = READING_DATA;
      }
      else
      {
        currentState = WAITING_FOR_AA;
      }
      break;
    case READING_DATA:
      static byte count = 0;
      if (Serial.available())
      {
        data[count++] = Serial.read();
      }
      if (count >= dataLength)
      {
        currentState = PROCESSING_DATA;
      }
    case PROCESSING_DATA:
      //code here to interpret data in the array
      currentState = WAITING_FOE_AA;
      break;
  };
}

I have lost track of tHE number of bytes of data so set dataLength to the appropriate value

you write faster than I :slight_smile:

Sir, I have edited the code that you sent me according to the knowledge gathered. The Packet data acquisition rate is massively high and I'm beginning to doubt myself. I know that the baud rate is high but errors can happen. So please be kind enough to share your opinion. I hope this article may help all Arduino users.

enum states
{
  WAITING_FOR_AA,
  WAITING_FOR_55,
  READING_DATA,
  PROCESSING_DATA
};
byte currentState = WAITING_FOR_AA;
const byte dataLength = 8;
byte data[dataLength];
void setup()
{
  Serial1.begin(115200);
  Serial.begin(115200);
}
int k;
void loop()
{ 
 /* if (Serial1.available())
      {
        k = Serial1.read();
      }
   Serial.println(k); */

  switch (currentState)
  {
    
    case WAITING_FOR_55:
      if (Serial1.read() == 0x55)
      {
       
        currentState = WAITING_FOR_AA;
      }
      break;
    case WAITING_FOR_AA:
      if (Serial1.read() == 0xAA)
      {
        currentState = READING_DATA;
        Serial.println("packet header aquired");
      }
      else
      {
        currentState = WAITING_FOR_AA;
      }
      break;
    case READING_DATA:
      static byte count = 0;
      if (Serial1.available())
      {
        data[count++] = Serial1.read();
      }
      if (count >= dataLength)
      {
        currentState = PROCESSING_DATA;
      }
    case PROCESSING_DATA:
      //code here to interpret data in the array
      currentState = WAITING_FOR_AA;
      break;
  }
}

So after modifying, I checked the data bytes and header bytes. but found a result like this and I don't know where got wrong.

the edited code for testing is as below.

enum states
{
  WAITING_FOR_AA,
  WAITING_FOR_55,
  READING_DATA,
  PROCESSING_DATA
};
byte currentState = WAITING_FOR_AA;
const byte dataLength = 8;
byte data[dataLength];
void setup()
{
  Serial1.begin(115200);
  Serial.begin(115200);
}
int k;
void loop()
{ 
if (Serial1.available())
      {
        k = Serial1.read();
      }
   Serial.println(k,HEX); 
delay(300);
  switch (currentState)
  {
    
    case WAITING_FOR_55:
      if (Serial1.read() == 0x55)
      {
       
        currentState = WAITING_FOR_AA;
      }
      break;
    case WAITING_FOR_AA:
      if (Serial1.read() == 0xAA)
      {
        currentState = READING_DATA;
        Serial.println("packet header aquired");
      }
      else
      {
        currentState = WAITING_FOR_AA;
      }
      break;
    case READING_DATA:
      static byte count = 0;
      if (Serial1.available())
      {
        data[count++] = Serial1.read();
      }
      if (count >= dataLength)
      {
        currentState = PROCESSING_DATA;
      }
    case PROCESSING_DATA:
      //code here to interpret data in the array
      currentState = WAITING_FOR_AA;
      break;
  }
}

I'm learning many from this forum and I'm really grateful to you both.

Why are you reading from Serial1 outside of the switch/case state machine ?

The whole point of using a state machine is that only code relevant to the current state is executed but you are reading a byte if it is available, no matter what state the system is in, printing it then ignoring it.

Suppose that byte is 0x55 and the system is in state WAITING_FOR_55 then the possible start of the header bytes will not be noticed and acted upon. What good is that ?

Oh.. my bad sir. is there any suggestions to demonstrate the data. Thank you very much for your valuable comments and time.

You were told not to insert extra lines in loop outside of the state machine - your code will not work with them.

try this code ( please do not edit )

enum states
{
  WAITING_FOR_AA,
  WAITING_FOR_55,
  READING_DATA,
  PROCESSING_DATA
};
byte currentState = WAITING_FOR_AA;
const byte dataLength = 8;
byte data[dataLength];
void setup()
{
  Serial1.begin(115200);
  Serial.begin(115200);
}
int k;
void loop()
{ 
/* ==== do not uncomment this lines  ====
if (Serial1.available())
      {
        k = Serial1.read();
      }
   Serial.println(k,HEX); 
delay(300); */
  switch (currentState)
  {
    
    case WAITING_FOR_55:
      if (Serial1.read() == 0x55)
      {
       
        currentState = WAITING_FOR_AA;
      }
      break;
    case WAITING_FOR_AA:
      if (Serial1.read() == 0xAA)
      {
        currentState = READING_DATA;
        Serial.println("packet header aquired");
      }
      else
      {
        currentState = WAITING_FOR_55:
      }
      break;
    case READING_DATA:
     {
      static byte count = 0;
      if (Serial1.available())
      {
        data[count++] = Serial1.read();
      }
      if (count >= dataLength)
      {
        currentState = PROCESSING_DATA;
        count = 0;
      }
      break:
     }
    case PROCESSING_DATA:
      //code here to interpret data in the array
      for ( byte i=0; i< dataLength; i++)  Serial.println( data[i[ ,HEX); 
     // Serial.println();
      currentState = WAITING_FOR_AA;
      break;
  }
}

Thank you very much, sir, but I need to show the full packet of data (with the header). However, I guess the editing is a necessary step cause there are many formulas that need to add here apart from the state machine. Are there any tutorials that show state-machines applications? however, acquiring and saving process is done and I'm really grateful to you both. I have learned many things.

I deliberately named the states WAITING_FOR_55 and WAITING_FOR_AA in the example to make it obvious what the states were doing and hard coded the corresponding values that the states were waiting for. You could, of course, name the states something like WAITING_FOR_HEADER_1 and WAITING_FOR_HEADER_2 and use variables for the values being looked for

That would make the state machine more general purpose and allow it to be used for many different message types

@ravirulz
Is the code from post#16 works for you?

of course
When I ask you do not edit the code, I just want that you tested exactly the code what I wrote. Since this code works (I hope?) you can add anything what you want to it.

In this example code your header bytes are always the same, so you can add printing constants to the PROCESSING_DATA case:

case PROCESSING_DATA:
      //code here to interpret data in the array
      Serial.println(0x55, HEX);
      Serial.println(0xAA, HEX);
      for ( byte i=0; i< dataLength; i++)  Serial.println( data[i[ ,HEX); 
      currentState = WAITING_FOR_AA;
      break;

If you don't get an AA when you are waiting for the AA you should go back to WAITING_FOR_55, not stay in WAITING_FOR_AA!