How to react to specific byte sequence over serial?

Projects General Guidance
1

Hello there,

I have a project where I will be monitoring a serial data line that I have zero control over. This data line sends regular messages of varying lengths at regular intervals, and my Arduino Mega could join the "call" at any point, thus not knowing if the bytes coming in are actually the start of a sequence or the middle of one. This serial data line is standard 9600 baud 8N1, and the only rules are that the first two bytes of any message constitute the device ID and the message length, while the final byte is always a checksum byte. However, these three bytes are not exclusive, and as such can appear in the middle of a message (albeit the chances of those three being present in a sequence that would have the checksum byte work out mid-message is approximately zero).

My project looks to listen for an incredibly specific message on said data line, and then immediately respond with a specific sequence of bytes if this message is detected, then idle and wait for the next time said specific message is received. For reference, the message I intend to react to will always look like this:

75 60 {any byte} {any byte} {any byte} {any byte} {any byte} {checksum}

The parts marked {any byte} could be just that; any byte. And as a result, the checksum will fluctuate based on the contents of those bytes. The checksum is a standard two's complement. So as an example of a complete sequence, this could potentially be an incoming message:

75 60 0A 57 60 00 FC 6E

So essentially, the first two bytes of the message will never change, the length of this specific message will always be 8 bytes, the final byte will always be a checksum, but the values of these bytes can appear within a message, and the Arduino could potentially join the bus mid-transmission, so a simple check for a start byte is no good, especially since other messages transmitted by devices on the bus may not be 8 bytes long.

Is there a best practice here? I'm afraid all the searches I've come up with don't appear to touch on this particular issue. I'm particularly worried about speed; the reply to this transmission must happen within 200 milliseconds of receipt or it will be considered "missed" by the device that transmitted it.

Thank you!

2

A couple of thoughts:

A) If the inter-message gap always exceeded a certain amount of time, then you could use that as a way of detecting the start of a message and then looking for 75 60 as the first 2 received bytes.

B) You could have a simple buffer (in your case 8 bytes) that continuously receives bytes and once it fills up, remove the oldest byte, shift the rest along 1 and add the new byte to the end. Keep looking for 75 60 as the first 2 bytes in the buffer and when you find it, compute the checksum of the received bytes. If you get a match to the received checksum, then you've got a message.

2 Likes
3

Have you read Serial Input Basics - updated ? If not then it's a good place to start.

Why do you think that? Please explain more, I suspect I know the answer but I want you to tell me (maybe I'm wrong).

Thanks,

4

I think that because the Arduino is an external device to the bus. The bus is running all of the time within a piece of equipment, and I am then hooking up my Arduino Mega while it is in the middle of operation and am essentially pretending to be the device on the bus that this message is intended for. That is why I mentioned earlier that I have zero control over the serial bus. I can only control the Arduino. There is no way to bring the Arduino and the bus up at the same time, and even if there were, I would still want the Arduino to be robust enough to handle being out of sync for a message or two before latching on.

I have read the serial input basics, but when I got to the juicy part about bytes, it said that what I was trying to do was out of scope, and the example it linked to didn't really seem to apply to what I was trying to achieve.

Thanks!

This is actually a pretty neat idea. I'll have to try it out, see if it's fast enough. Should be on the Mega.

5

You can avoid the shifting part by keeping track of the head and tail of the queue of bytes received. That would take less time as you're not copying bytes around.

6

I don't see that as a problem. You connect the Arduino (or power it up) and whatever is on the bus is on the bus, the Arduino starts looking for the particular sequence, when the sequence comes along it detects it and does its thing.

My guess at what you were going to say was wrong however. What I now need from you is to know why you think what you want to do would be a problem because from my point of view it's not and I can't think why you would think it is. Just write code to look for the sequence, when the code sees the sequence it will do its thing.

I don't know if this will help you or confuse you but have a look at: Using Nextion displays with Arduino and look how the code works. Obviously I realise the code is not for what you are doing but the principals apply, the code looks for a particular sequence from a Nextion display and reacts accordingly. Although you might expect the Nextion and the Arduino to be powered at the same time it does not really matter; as long as they are connected and the Arduino code is running then it will work regardless of when they were each powered up and connected to each other.

1 Like
7

It is done as @markd833 wrote. Shifting the data in a loop or with memmove() or keeping track of the head (and tail), that is all the same to me.

Suppose there is a 0x75 0x60 in the middle of a message and suppose the byte that is read as a checksum happens to be the right checksum, then it will accept the wrong data. That is possible, since the protocol is not 100% secure.
I'm not good at math. What is the chance that this will happen ? I guess once in 10 million.
Such thing happen more often when the line is open and there is random noise on the line.

1 Like
8

Simply because it's entirely possible for a serial message on the bus to contain '75 60' as bytes adjacent to one another in the middle of a message. There's no rule against it. So if I just lock on based on those two bytes, I could miss the actual message. For example, take this potential stream of data from the bus:

...57 75 60 C2 75 60 A7 0F 0C EB 75 09...

I join the bus "late," and end up in the middle of a previous message. I get my 75 60 and 8 bytes of data, and within that I also get another 75 60. Now, obviously the checksum won't match on the first 75 60 and will on the second, because the second is the correct message. But I have to account for that potential situation is why I specifically point out that joining late could be a problem.

@Koepel, you beat me to it, but I'll post this anyway. The method @markd833 wrote in "B" seemed like the way to go, so I'm going to work on the code today and see about benchmarking its speed. Once I have more to show, I'll go ahead and post back and if it just plain works, I'll mark the solution. For now, thank you all for such speedy responses!

9

OK, thanks, I understand now, I've re-read your OP and I guess you said it there but I missed it. In that case what @markd833 said in reply #2 covers it; look for the gap between messages and sync from there.

1 Like
10

Unless you write the worst code I have ever seen on here its speed will be a lot faster then the serial port so more than fast enough.

1 Like
11

I think I can put some stress on the sketch with a sender sketch.
It is not tested and I don't understand the checksum.

// Create packages of 8 byte
// 0x75, 0x60, 0x??, 0x??, 0x??, 0x??, 0x??, checksum
// The checksum is a 2's complement ?

// Set _SERIAL_ to the used serial bus.
#define _SERIAL_ Serial

unsigned long counter;

void setup() 
{
  _SERIAL_.begin( 9600);    // 9600, 8N1
}

void loop() 
{
  int x = random( 10);

  if( x == 0)
  {
    // send wrong data, with right checksum
    byte buf12[12];
    buf12[0] = 0x75;
    buf12[1] = 0x60;
    buf12[2] = 0x75;
    buf12[3] = 0x60;
    buf12[4] = 0x75;
    buf12[5] = 0x60;
    buf12[6] = (byte) random( 256);
    buf12[7] = checksum( buf12);
    buf12[8] = (byte) random( 256);
    buf12[9] = checksum( &buf12[2]);
    buf12[10] = (byte) random( 256);
    buf12[11] = checksum( &buf12[4]);
    _SERIAL_.write( buf12, sizeof( buf12));
  }
  else if( x == 1)
  {
    // send data with wrong checksum
    byte buf[8];

    buf[0] = 0x75;
    buf[1] = 0x60;
    buf[2] = (byte) random( 256);
    buf[3] = (byte) random( 256);
    buf[4] = (byte) random( 256);
    buf[5] = (byte) random( 256);
    buf[6] = (byte) random( 256);
    buf[7] = (byte) (checksum( buf) + 1 + random( 250));  // to be sure the checksum is wrong
    _SERIAL_.write( buf, sizeof( buf));
  }
  else if( x == 2)
  {
    // send totally wrong data
    int n = random( 1, 50);
    for( int i=0; i<n; i++)
    {
      _SERIAL_.write( (byte) random( 256));
    }
  }
  else
  {
    // send valid data
    byte buf8[8];

    buf8[0] = 0x75;
    buf8[1] = 0x60;
    buf8[2] = (byte) (counter >> 24);   // counter value (for valid data only)
    buf8[3] = (byte) (counter >> 16);
    buf8[4] = (byte) (counter >> 8);
    buf8[5] = (byte) counter;
    buf8[6] = (byte) random( 256);      // just a spare byte
    buf8[7] = checksum( buf8);
    _SERIAL_.write( buf8, sizeof( buf8));
    counter++;
  }

  // random gap, max 50ms, 50% chance no gap
  if( random( 2) == 0)
  {
    delay( random( 50));
  }
}


byte checksum( byte data[])
{
  byte total = 0;

  for( int i=0; i<7; i++)
  {
    total += data[i];
  }
  byte chk = (byte) (256 - (int) total);
  return( chk);
}

It is possible to use Wokwi in Chrome and connect to a serial port of the computer (in development). That is not in my simulation yet. This is just the sketch in Wokwi: https://wokwi.com/arduino/projects/310250258016764481.

[ADDED] Is it okay to put a non-working sketch here ?
I can't find a second Arduino board and it is it not working in Wokwi and I don't have time today to check it. This is my sketch so far:

//
// Warning: Not working yet !
//
// Special Wokwi settings:
//   Use this in the Chrome browser.
//   The Serial input is from a real serial port on the computer !
//   For example the real device or a real Arduino board.
//   The Serial output is redirected to the simulated Serial Monitor
//   I tried to move everything to 115200, but that dit not work either.
//
//  MysterySerialReceiver.ino   (this sketch)
//    https://wokwi.com/arduino/projects/310261647915614785
//  MysterySerialSender.ino
//    https://wokwi.com/arduino/projects/310250258016764481
//
//
// Receive packages of 8 byte
// 0x75, 0x60, 0x??, 0x??, 0x??, 0x??, 0x??, checksum
// The checksum is a 2's complement ?

// Set _SERIAL_ to the used serial bus.
#define _SERIAL_ Serial

byte buffer[8];
int index = 0;

void setup() 
{
  _SERIAL_.begin( 9600);     // 9600, 8N1
}

void loop() 
{
  if( _SERIAL_.available() > 0)
  {
    buffer[index] = (byte) Serial.read();
    Serial.print( buffer[index], HEX);
    index++;

    if( index == 8)          // received eight bytes ?
    {
      if( buffer[0] == 0x75 && buffer[1] == 0x60 && checksum( buffer, sizeof(buffer) == 0))
      {
        // The data is valid
        Serial.println( "New Data ");
        for( int i=0; i<8; i++)
        {
          Serial.print( "0x");
          if( buffer[i] < 0x10)
            Serial.print( "0");
          Serial.print( buffer[i], HEX);
          Serial.print( ", ");
        }
        // Create the counter from the MysterySerialSender.ino sketch.
        uint32_t counter;
        counter = (uint32_t) buffer[5];
        counter |= ((uint32_t) buffer[4]) << 8;
        counter |= ((uint32_t) buffer[3]) << 16;
        counter |= ((uint32_t) buffer[2]) << 24;
        Serial.print( " (");
        Serial.print( counter);
        Serial.print( ")");
        Serial.println();

        index = 0;           // clear the buffer, use the buffer from the beginning
      }
      else
      {
        // Shift everything one location to the left and check the data
        // the next time that the loop() runs and something is received.
        for( int i=0; i<7; i++)
          buffer[i] = buffer[i+1];
        index = 7;           // there is new free spot at the end of the buffer
      }
    }
  }
}

byte checksum( byte data[], int len)
{
  byte total;

  for( int i=0; i<len; i++)
  {
    total += data[i];
  }
  byte chk = (byte) (256 - (int) total);
  return( chk);
}
12

Okay, I think I have some code that will work. This is a super simplified version and doesn't include the response part, but I already know how to talk on Serial so that's not as much of a problem as reading and confirming a message. So here's my snippet:

byte window[8];

void setup() {
    Serial.begin(9600);
}

void loop() {
    if (Serial.available()) {
        for (uint8_t i = 0; i < 7; i++) {
            window[i] = window[i + 1];
        }
        window[7] = Serial.read();
        if ((window[0] == 0x75) && (window[1] == 0x60)) {
            byte checksum = 0;
            for (uint8_t i = 0; i<7; i++) {
                checksum += window[i];
            }
            checksum = 0xFF - checksum;
            checksum += 0x01;
            }
            if (checksum == window[7]) {
            }
        }
    }
}

Hopefully that works. I won't actually be able to test it in-situ for a day or two as I have to create the correct serial interface to connect to the equipment. I think that'll be faster than running a test program on a second Arduino, at least for now. Again, thank you all for your assistance thus far, and hopefully my next post will be success!

1 Like
13

I wanted to post an update here to give the thread some closure; don't you just hate it when there's a thread that just ends with a cliffhanger?

The test code above worked a treat. The sliding window correctly detects the required message, and by just poisoning the window with window[0]=0x00 when my response code runs I can be absolutely sure that my response code will only run once per received message.

Thank you all!

1 Like
14

Thank you, yes, I hate topics that die without a resolution or anything. Thanks for the update and glad you got it working.

15

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.