Is there a way to read parts of incoming serial?

Hello!

I’m writing on my bachelor’s degree in electrical engineering and I will be receiving a hexadecimal string which has a start byte but no stop byte. It will look something like: |CC|45|01|23|4A|F2| where the last two bytes are CRC (MSB and LSB). The 3rd byte is the number of following message bytes.

My issue is that I need to recieve this string not knowing when it really ends unless I calculate it from the 3rd byte that tells me how long the rest of the string will be.

Is there a way to read 3 bytes from the serial buffer, do the calculation for how much more to read, and then continue to read from the buffer? I’m thinking something like this where I can first read the first 3 bytes and then calculate and then keep reading, but I cannot get it to work.

char c[10];

    if (Serial1.available()) {                                                                       
      
      int i;
      for(i = 0; i < 3; i++) 
        c[i] = Serial1.read();                                                             
               
      while (Serial1.available() > 0 ) {
          c[i] = Serial1.read();
          i++;
      }
    }

Your example program uses Serial1.read() without using Serial1.available() so it may "read" characters that have not arrived yet.

Your Arduino is MUCH faster than the serial interface so, Yes, you can read three characters and calculate how many more to read - BUT you have to do it correctly.

Your code has flaws

If Serial1.available() returns true how do you know that the complete message is available ?

Assuming that it is available you read 3 bytes but do not attempt to use the data to determine the full length of the message. Instead you simply read bytes until no more are available. To compound things further the array is declared to hold a maximum of 10 characters. What will happen if you receive more than 10 ? Even your example message has more than 10 characters.

How about waiting until at least 3 bytes are available then reading 3 bytes and interpreting the third one as the length of the whole message before reading it ?

There are snags with this. For instance, what is the maximum length of the message ?

Do you have any control over the format of the message ?

Another option is to treat the start-marker as an end-marker. It means that you lose the first message but after that you should have them all.

...R
Serial Input Basics - simple reliable ways to receive data.

UKHeliBob:
Your code has flaws

If Serial1.available() returns true how do you know that the complete message is available ?

Assuming that it is available you read 3 bytes but do not attempt to use the data to determine the full length of the message. Instead you simply read bytes until no more are available. To compound things further the array is declared to hold a maximum of 10 characters. What will happen if you receive more than 10 ? Even your example message has more than 10 characters.

How about waiting until at least 3 bytes are available then reading 3 bytes and interpreting the third one as the length of the whole message before reading it ?

There are snags with this. For instance, what is the maximum length of the message ?

Do you have any control over the format of the message ?

Thank you for helping out! I have modified my code after your input about waiting until the message byte is actually recieved. It does look like it’s gonna work. But I have one problem and that is the case where there will be no message. Because it seems like arduino read “00” aka NULL and then just terminates it or ignores it so I can never get NULL to be passed into the buffer it seems. (Can I use an int array instead?)

I have no control over the format of the String but I do know the maximum length it can have, I have updated the buffer.

#define ArduinoIDE Serial
#define RS422 Serial4          

char buffer[262];               // hex-string buffer
int msgLen;                     // length of message
bool receivedString = false;    // check received full string
bool recvMsgLen = false;        // check received message length


void setup() {
  
  // begin serial communication
  ArduinoIDE.begin(9600);  // serial monitor 
  RS422.begin(9600);       // this will be the actual incoming data later on

  // wait for monitor to open
  while(!ArduinoIDE){}

  // print startup monitor
  ArduinoIDE.print("Sketch:   ");   ArduinoIDE.println(__FILE__);
  ArduinoIDE.print("Uploaded: ");   ArduinoIDE.print(__DATE__);   ArduinoIDE.print(", ");   
  ArduinoIDE.println(__TIME__);     ArduinoIDE.println(" ");

}



void loop() {

  // if first three bytes has arrived, read them to buffer-array
  
  if (ArduinoIDE.available() >= 3 && !recvMsgLen) {
    for (int i=0; i<3; i++) 
      buffer[i] = Serial.read();    // read [sync], [command], [msg length] -bytes
    msgLen = buffer[2];             // save length of message as int
    recvMsgLen = true;              // set received to true so this code only executes once
  }


  // if message-byte was saved to the buffer and it's 2 or more
  
  if (recvMsgLen && msgLen >= 2 && !receivedString){
    
    // read until last CRC bit, calculated from message length -byte
    if (ArduinoIDE.available() >= msgLen+1) {
      for (int i = 1; i <= msgLen+1; i++)   // msgLen +1 because the first message-byte is always NULL
          buffer[i+2] = Serial.read();      // save the message and CRC bytes to buffer 
      receivedString = true;
    }
  }
  else if (msgLen == 0){  // this is the case where there is no message aka "length byte = null"
    //                               |sync|cmd | CRC |  <--- no length byte between cmd and CRC because it's NULL
    // not sure how to solve this if | AA | 11 |F2|D2|
  }



  // print buffer array to serial monitor
  
  if (receivedString){
    for (int i=0; i<3; i++)
      ArduinoIDE.print(buffer[i], HEX);
    
    ArduinoIDE.print('\n');
    receivedString = false;        // reset so we can receive more messages
    recvMsgLen = false;             //         ---   "   ---
    
    // reset buffer array somehow here
  }
}

Is there a noticeable time gap between the end of one message and the start of the next?

If so you can use that gap as the end-marker by checking the interval since the most recent time when there was data in the Serial Input Buffer.

...R

I will be receiving a hexadecimal string which has a start byte but no stop byte.

Are you absolutely sure that the message is not followed by a Carriage Return, Linefeed or maybe both ?

What device is sending the message ?

Robin2:
Is there a noticeable time gap between the end of one message and the start of the next?

If so you can use that gap as the end-marker by checking the interval since the most recent time when there was data in the Serial Input Buffer.

...R

Thank you for your inputs, I did also consider using the start marker as an end marker. There is no significant time between, it's all gonna happen very fast and asynchronously!

UKHeliBob:
Are you absolutely sure that the message is not followed by a Carriage Return, Linefeed or maybe both ?

What device is sending the message ?

Yes I am very sure about this! I can't go into detail on my project due to the nature of it.

Marius1992:
Thank you for your inputs, I did also consider using the start marker as an end marker.

And why have you discarded the idea?

If the data is happening fast that would seem to me to be a good solution.

...R

Robin2:
And why have you discarded the idea?

If the data is happening fast that would seem to me to be a good solution.

...R

I discarded it because I need to detect every message and I can't discard the first one.

@OP

|CC|45|01|23|4A|F2|

1. Is the above one is an example of a complete frame that is coming down to Arduino, then my questions are:

(1) What are these 'bar signs/symbols' among the bytes?

(2) Is CC is the 'Mark of Start' of a frame?

(3) The 3rd byte (01) (counting from left) indicates the number of 'data byte' in the frame is one, and it is this byte -- 23..

(4) As you have said that 4AF2 is the CRC -- fine; it is not CHKSUM?

(5) What is this byte -- 45 (counting from left)? What is the meaning of this field?

(6) Are the digits of the frame transmitted as: 434344453031323334414632 (each byte corresponds to the 80bit ASCII Code of the corresponding hex-digit of the given frame)? Or, are they transmitted as 8-bit binary as they are shown in the example?

2. BTW: If CC is the 'Mark of Start', then we must be sure that this CC will never appear in the frame as a data byte. Probably, you can't be sure; so, there is enough doubt that CC can be candidate for a 'Mark of Start'. Please, post a complete frame containing multi-byte data; hopefully, you will find an amicable solution.

Marius1992:
I discarded it because I need to detect every message and I can't discard the first one.

That sounds reasonable until I ask myself how would I know which is the first one?

As a separate comment, I think it would not be difficult to modify my example that uses an end-marker so that it saves (and delivers to you) all the characters it receives up to the second "end-marker" which would really be the second occurrence of the start-marker. From that point on it can work as normal.

...R

You can do this with a statemachine but you/we need to know more about the protocol.

For example, in your OP you mention that each message has a start byte and your example showed 0xCC. You also mention that the 3rd byte describes the number of data bytes to follow.

But in post #4 you show an example of AA:11:F2:D2 where the 3rd byte does not describe the number of bytes in the payload and the start byte is not 0xCC.

This implies different start bytes for different message types. A header of 0xCC seems to indicate messages that expect at least one byte of data whereas messages with a header of 0xAA do not.

That's cool and easily solved but in order to do so, you have to describe all possible header types and the structure of the message that follows.

What are all the header types?
What are all the commands for each header type?
What are the expected parameters for commands in messages that expect data?

To solve this problem you/we need as much info as you can provide.

Blackfin:
This implies different start bytes for different message types. A header of 0xCC seems to indicate messages that expect at least one byte of data whereas messages with a header of 0xAA do not.

If the system is complex in that way I would expect there to be some other simple means to identify the start or end of a message.

Alternatively the system designer needs to go back to school.

@Marius1992, please post a link to the datasheet that defines the system you are trying to work with.

...R

I have solved this issue now everybody! Thank you all for your inputs, it was very helpful.

Marius1992:
I have solved this issue now everybody! Thank you all for your inputs, it was very helpful.

I would be interested in seeing your solution

Here is my code for my solution. The problem that I had that I could not read “0x00”'s because they terminated the char-array. I solved with using int-array. I simulate a RS-422 signal with Docklight that comes in over serial. I then check if the buffer on that serial port contains the sync-byte, if it does I read the rest of the buffer and calculate the length that I have to read to reach the end of my message since there was no terminating character. The code also shows how I print out to the serial monitor and how I send the information over bluetooth to be received by an iOS app.

I use the Teensy 3.6, this is why I can use several Serial-ports.

#define ArduinoIDE Serial
#define RS422 Serial4   
#define IOS Serial3 

int buffer[262];                // hex-string buffer
int msgLen;                     // length of message
int syncByte = 0;
bool receivedString = false;    // check received full string
bool recvMsgLen = false;        // check received message length

void setup() {
  
  // begin serial communication
  ArduinoIDE.begin(9600);  // serial monitor 
  RS422.begin(9600);  // this will be the actual incoming data later on
  IOS.begin(9600);         // start bluetooth serial communication
  // wait for monitor to open
  while(!ArduinoIDE){}

  // print startup monitor
  ArduinoIDE.print("Arduino is ready!\n");
}



void loop() {

  receiveRS422();
  sendToApp();
  printReceivedToIDE();
 
}

void receiveRS422(){
  // if first three bytes has arrived, read them to buffer-array
  
  if (RS422.available() >= 3 && !recvMsgLen) {
    syncByte = RS422.read();           // read possible sync byte
        
    if (syncByte == 204){              // does syncByte equal "CC"? 
      buffer[0] = syncByte; 
      for (int i=1; i<3; i++) 
        buffer[i] = RS422.read();      // read [command], [msg length] -bytes   
    
      msgLen = buffer[2];              // save length of message as int
      recvMsgLen = true;               // set received to true so this code only executes once
    }
  }


  // if message-byte was saved to the buffer and there is a message
  // *** only executes if sync-byte was detected and the first 3 bytes saved to buffer***
  
  if (recvMsgLen && msgLen > 0 && !receivedString){
    
    // read until last CRC bit, calculated from message length -byte
    if (RS422.available() >= msgLen+2) {
      for (int i = 1; i <= msgLen+2; i++)       // 
          buffer[i+2] = RS422.read();      // save the message and CRC bytes to buffer 
      receivedString = true;
    }
  }


  // if message-byte was saved to the buffer and there is no message
  // ** only executes if sync-byte was detected and the first 3 bytes saved to buffer**
  
  else if (recvMsgLen && msgLen == 0 && !receivedString){  
    if (RS422.available() >= 2) {
      for (int i = 1; i <= 3; i++)       // 
          buffer[i+2] = RS422.read();      // save the message and CRC bytes to buffer 
      receivedString = true;
    }
  }
}

void printReceivedToIDE(){       // print data to IDE
  
  if (receivedString) {               // if there was a full string received
      ArduinoIDE.println("\nHexadecimal string received: ");
      
      if (msgLen > 0){                // print for cases where the message is larger than zero
        for (int i=0; i < msgLen+5; i++){
          ArduinoIDE.print("[");
          ArduinoIDE.print(buffer[i], HEX);
          ArduinoIDE.print("]");
        }
        for(int i = 0; i <= msgLen+5; i++)  // reset buffer values
          buffer[i] = 0;
      }
      else if (msgLen == 0) {         // print for cases where there is no message
        for(int i=0; i <= 4; i++) {
          ArduinoIDE.print("[");
          ArduinoIDE.print(buffer[i], HEX);
          ArduinoIDE.print("]");  
        }  
        for(int i = 0; i <= 4; i++)   // reset buffer values 
          buffer[i] = 0;
      }
      
      ArduinoIDE.print('\n');
      receivedString = false;         // reset so we can receive more messages
      recvMsgLen = false;             //         ---   "   ---
     
  }  
}

void sendToApp(){           // send data to app over bluetooth
  if(receivedString){
    for(int i = 0; i <= msgLen+4; i++)
      IOS.write(buffer[i]);
  }   
}

This is fine; but, you failed to post a complete frame in response to Post#11. If you could do that (or even if you do it now), I could make an attempt to design a program that would receive the said frame.

A couple of observations

The problem that I had that I could not read “0x00”'s because they terminated the char-array

Whilst it is true that a value of 0 terminates a C style string when the array of chars is used as a string you do not need to use it that way and are free to use the data in the array of chars as you choose. In any case, why use an int when a byte will do. Use the smallest data type that fits the needs of the program and save memory.

In a similar vein, msgLen is declared as an int but the maximum value that it will hold is 255 as it is read from the serial so a byte would do.

Which brings me to the fact that the message buffer is declared as having 262 elements but msgLen will have a maximum value of 255.

   if (syncByte == 204)               // does syncByte equal "CC"?

Why use a “magic” number when you could have declared a constant ? In addition, putting CC in quotes rather than using 0xCC makes it look you are testing for a String rather than a Hex value

      for (int i = 1; i < 3; i++)
        buffer[i] = RS422.read();      // read [command], [msg length] -bytes
      msgLen = buffer[2];              // save length of message as int

buffer[2] will contain an int but its maximum value will be 255

I am sorry if my observations seem harsh and you may have reasons for writing the program in the way that you did but there do seem to me to be some oddities with it.