[SOLVED] What if a terminator character can also be part of the string?

Eddiie · November 20, 2019, 12:46am

Hi all,

Today I am hacking on some device using serial. I'd like to put the incoming bytes into a string. Looking at the stream of data, it appears that 0x55 is the end of string signal. However, I also see streams where 0x55 appears two times.

Come to think, 0x55 is actually an ASCII character (7) so it could appear in the middle of a stream.

How would I check / confirm that the 0x55 it receives is indeed the end of packet?

Would it be as simple as checking the serial port for more bytes (bits?)? For example, is this a good idea?

if (inChar == '0x55' && Serial.available() == 0) { ... }

This would confirm the terminator was hit and there is no more data? Seems too simple.

Appreciate the input.

Idahowalker · November 20, 2019, 12:54am

Here is a task that I have used to read the serial port using start of sentence and end of sentence characters:

void fReceiveSerial_LIDAR( void * parameters  )
{
  bool BeginSentence = false;
  char OneChar;
  char *str;
  str = (char *)ps_calloc(300, sizeof(char) ); // put str buffer into PSRAM
  log_i("Free PSRAM before String: %d", ESP.getFreePsram());
  for ( ;; )
  {
    EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
    if ( LIDARSerial.available() >= 1 )
    {
      while ( LIDARSerial.available() )
      {
        OneChar = LIDARSerial.read();
        if ( BeginSentence )
        {
          if ( OneChar == '>')
          {
            if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
            {
               xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &str );
              xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
              //
            }
            BeginSentence = false;
            break;
          }
          strncat( str, &OneChar, 1 );
        }
        else
        {
          if ( OneChar == '<' )
          {
            strcpy( str, ""); // clear string buffer
            BeginSentence = true; // found beginning of sentence
          }
        }
      } //  while ( LIDARSerial.available() )
    } //if ( LIDARSerial.available() >= 1 )
    xSemaphoreGive( sema_ReceiveSerial_LIDAR );
    //        log_i( "fReceiveSerial_LIDAR " );
    //        log_i(uxTaskGetStackHighWaterMark( NULL ));
  }
  free(str);
  vTaskDelete( NULL );
} //void fParseSerial( void * parameters  )

johnwasser · November 20, 2019, 1:10am

      if (inChar == '0x55' && Serial.available() == 0)
       {
         ... 
      }

You mean "inChar == 0x55" but yes, that would indicate that there are no characters in the buffer after the 0x55. You can't be exactly sure that you didn't catch the moment between the 0x55 arriving and another character arriving.

Perhaps there is a header with a length field? That would allow you to identify the last character.

gfvalvo · November 20, 2019, 1:27am

Serial transmission protocols often use a known frame length (either fixed or specified in header) or minimum inter-frame idle gap that can be identified.

Eddiie · November 20, 2019, 1:55am

Idahowalker:
Here is a task that I have used to read the serial port using start of sentence and end of sentence characters:
void fReceiveSerial_LIDAR( void * parameters  )

{
bool BeginSentence = false;
char OneChar;
char *str;
str = (char *)ps_calloc(300, sizeof(char) ); // put str buffer into PSRAM
log_i(“Free PSRAM before String: %d”, ESP.getFreePsram());
for ( ;; )
{
EventBits_t xbit = xEventGroupWaitBits (eg, evtReceiveSerial_LIDAR, pdTRUE, pdTRUE, portMAX_DELAY);
if ( LIDARSerial.available() >= 1 )
{
while ( LIDARSerial.available() )
{
OneChar = LIDARSerial.read();
if ( BeginSentence )
{
if ( OneChar == ‘>’)
{
if ( xSemaphoreTake( sema_ParseLIDAR_ReceivedSerial, xSemaphoreTicksToWait10 ) == pdTRUE )
{
xQueueOverwrite( xQ_LIDAR_Display_INFO, ( void * ) &str );
xEventGroupSetBits( eg, evtParseLIDAR_ReceivedSerial );
//
}
BeginSentence = false;
break;
}
strncat( str, &OneChar, 1 );
}
else
{
if ( OneChar == ‘<’ )
{
strcpy( str, “”); // clear string buffer
BeginSentence = true; // found beginning of sentence
}
}
} // while ( LIDARSerial.available() )
} //if ( LIDARSerial.available() >= 1 )
xSemaphoreGive( sema_ReceiveSerial_LIDAR );
// log_i( "fReceiveSerial_LIDAR " );
// log_i(uxTaskGetStackHighWaterMark( NULL ));
}
free(str);
vTaskDelete( NULL );
} //void fParseSerial( void * parameters )

Wow, this is way over my head.

Eddiie · November 20, 2019, 2:18am

@johnwasser , @gfvalvo

I think I have found that the 4th byte sets the length of the message to follow.
But I am early in the understanding the packet structure, need more messages to study.
There must be a checksum somewhere too.

Here are some streams -

From host -
0xaa, 0x20, 0x35, 0x00, 0x55, 0x55

Device responds -
0xaa, 0x00, 0xff, 0x02, 0x20, 0x35, 0x56, 0x55

250ms later,

From host -
0xaa, 0x20, 0x01, 0x00, 0x21, 0x55

Device responds -
0xaa, 0x00, 0xff, 0x10, 0x20, 0x01, 0x4d, 0x45, 0x54, 0x52, 0x00, 0x56, 0x30, 0x35, 0x2e, 0x30, 0x31, 0x00, 0xc4, 0x02, 0x78, 0x55

Or shown n ASCII (/DEC?) -
170, 0, 255, 16, , 1, M, E, T, R, 0, V, 5, ., 0, 1, 196, 2, x, U

Which is indeed the device sayng it is Meter 0 with firmware v05.01

The 4th byte, I am claiming is the length. 16 characters fits with the number of chars after the 16.

Where this breaks down is message from the host... 4th byte is 0 and there are 2 bytes after.
It could be messages from the host are limited to 6 chars.
In fact, that could be it.

I am looking to emulate the 'device'. So just need to look for 6 bytes from the host.

Any thoughts on checksum, or packet structure from above is greatly appreciated!

MorganS · November 20, 2019, 2:45am

What is the device and is there any documentation on the protocol?

Without documentation, you just have to collect a lot of data like you’ve already done and mess with it until it works.

DrAzzy · November 20, 2019, 3:54am

If the terminator character can also be part of the string, then it's not a terminator character.

dougp · November 20, 2019, 4:07am

The opening and closing of the message strings is curious - AA and 55. XOR those together and get zero. Some weird parity/validation scheme?

Eddiie · November 20, 2019, 8:10am

The device is a dot matrix display by Adaptive. They made it for a third party, there is no documentation on the protocol, that would just be too easy.

Currently running data captures and trying to reply the data. But so far not luck.

Might be timing.

Also using the software serial port (38400 baud), not sure if that slows things down and the protocol is time sensitive or not.

It is RS485, converting it to RS232, I seem to be playing the bits back right.

Thank you all for the help.

Robin2 · November 20, 2019, 8:58am

Eddiie:
Today I am hacking on some device using serial. I'd like to put the incoming bytes into a string. Looking at the stream of data, it appears that 0x55 is the end of string signal. However, I also see streams where 0x55 appears two times.

A crude solution, which might work, or might just be useful for learning, is to use the second example in Serial Input Basics with 0x55 as the end-marker. It will receive the message as a series of chunks and you will have to join them together and reinstate the 0x55 character which will have been stripped out.

...R

johnwasser · November 20, 2019, 1:59pm

So it looks like:

0xaa
Two byte message type
N (length byte)
N data bytes
Checksum byte
0x55

I'm not sure how the checksum works. It appears to be the simple sum of the data bytes:
0x20+0x35+0x00==0x55
0x20+0x01+0x00==0x21
0x00+0xff+0x10+0x20+0x01+0x4d+0x45+0x54+0x52+0x00+0x56+0x30+0x35+0x2e+0x30+0x31+0x00+0xc4+0x02 == 0x78

That doesn't quite work for this message:
0x00+0xff+0x02+0x20+0x35 == 0x56 but the checksum is 0x36 (off by 0x20)
Are you sure you got all the bytes right?

If the header byte, length, checksum, and trailing byte all match, the message is valid.

Eddiie · November 20, 2019, 7:55pm

@johnwasser
Thank you for your time and input. I will study this more tonight.
It is hard to tell with the limited captures.
There are other devices, going try a different one soon.

I have another capture, this one puts the display in a diagnostic mode -

From host (6 bytes) -
0xAA, 0x20, 0x40, 0x00, 0x60, 0x55

Response from Device (0.00029 seconds later)
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x40, 0x61, 0x55

1.5 seconds later

From host -
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55

Response from Device (0.00287 seconds later) -
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

2.3 seconds later, ping pongs back and forth until diagnostics stopped..
Host - 0xFF
Device - 0xFF

...
This message repeats every 5 seconds

From Host -
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55

Response from Device -
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

.....
Stop Diag message (I think this is a stop diag message) -

Sequence of 5 0xFF from Host.

2.2 seconds later back to the loop of firmware check..
From Host -
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55

From Device -
0xAA, 0x00, 0x04, 0x02, 0x20, 0x42, 0x68, 0x55

From Host -
0xAA, 0x20, 0x35, 0x00, 0x55, 0x55

From Device -
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x35, 0x36, 0x55

From Host -
0xAA, 0x20, 0x01, 0x00, 0x21, 0x55

From Device -
0xAA, 0x00, 0xFF, 0x10, 0x20, 0x01, 0x4D, 0x45, 0x54, 0x52, 0x00, 0x56, 0x30, 0x35, 0x2E, 0x30, 0x31, 0x00, 0xC4, 0x02, 0x78, 0x55
....

Eddiie · November 20, 2019, 8:06pm

johnwasser:
That doesn't quite work for this message:
0x00+0xff+0x02+0x20+0x35 == 0x56 but the checksum is 0x36 (off by 0x20)
Are you sure you got all the bytes right?

My confirmation was wrong. John is right, this is a typo on the checksum byte.

Correct checksum should be 0x56, no 0x36

johnwasser · November 20, 2019, 10:56pm

Only two of these ten messages don't fit the pattern.

0xAA, 0x20, 0x40, 0x00, 0x60, 0x55
0xAA, 0x20, 0x42, 0x62, 0x42, 0x55 (length = 0x62???)
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55
0xAA, 0x00, 0x04, 0x02, 0x20, 0x42, 0x68, 0x55
0xAA, 0x20, 0x35, 0x00, 0x55, 0x55
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x35, 0x36, 0x55 (0x20 short of expected checksum)
0xAA, 0x20, 0x01, 0x00, 0x21, 0x55

Eddiie · November 21, 2019, 1:36am

@johnwasser

You're right! Going back to the source file those are typos.
Fixing the post.

I think you've figured out the checksum!

I dont understand how this works though -

From Device
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

Checksum = 163 but if you drop the 1 from 163, it matches. Seems like there is some bitwise something that happens?

Eddiie · November 21, 2019, 2:07am

Now for the first part of the packet

John proposes (removed last two bytes, checksum and end of transmission)-
Byte 1 = 0xaa
Byte 2 = Two byte message type
Byte 3 = N (length byte)
Byte 4 = N data bytes

We can ignore Byte 1 for now, leaving only a few bytes left in the header…

Bytes2, 3, 4

Looking at the packet header from the dislpay back to the host on the firmware messages,
0xAA, 0x00, 0xFF, 0x10, 0x20, 0x01, M, E, T, R, …

Byte 1 = 0xAA
Byte 2 = 0x20 (from host), 0x00 (from device), Seems to always be 0x00 or 0x20 Message type, intereting thought, Read / Write ? What types can there be?
Byte 3 = 0x35, 0x01 (from host, seems to alternate), always seems to be 0xFF from the device.
Byte 4 = Size of the packet, I am most certain of this

Byte 5 - Byte N - 2 is the message
Last 2 bytes = checksum and EoT

This is based on a RS485 multi-drop connection, so there can be multiple devices on this bus the host wants to talk to. Therefore, I think one of these packets should be a device id or address.

Thoughts?

dougp · November 21, 2019, 2:28am

Eddiie:
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

Checksum = 163 but if you drop the 1 from 163, it matches. Seems like there is some bitwise something that happens?

If you interpret 0xFF as a signed number (that being -1) it works.

dougp · November 21, 2019, 2:41am

Eddiie:
This is based on a RS485 multi-drop connection, so there can be multiple devices on this bus the host wants to talk to. Therefore, I think one of these packets should be a device id or address.

Thoughts?

Eddiie:
From host (6 bytes) -
0xAA, 0x20, 0x40, 0x00, 0x60, 0x55

Response from Device (0.00029 seconds later)
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x40, 0x61, 0x55

1.5 seconds later

From host -
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55

Response from Device (0.00287 seconds later) -
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

Notice the echoed sequences. Host sends start character then two address characters? Device responds with start character and whatever followed by its own address?

Eddiie · November 21, 2019, 3:42am

@dougp
The 0xFF being treated as a signed number... subtract 1 - Have you seen this done anywhere else before? - 0xff being treated as a signed number?

I like what you're saying about 0x20 and 0x42. It seems to only happen on the beginning packet from the Host -

Here it is on the firmware conversation -

From Host
0xAA, 0x20, 0x35, 0x00, 0x55, 0x55

From Device
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x35, 0x56, 0x55

Here are 2 from the Diagnostic Mode conversation -
From Host
0xAA, 0x20, 0x40, 0x00, 0x60, 0x55

From Device
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x40, 0x61, 0x55

2 second pause and it appears again

From Host
0xAA, 0x20, 0x42, 0x00, 0x62, 0x55

From Device
0xAA, 0xFF, 0x02, 0x20, 0x42, 0x63, 0x55

At some point I see the 0x20 0x35 again.

From Host
0xAA, 0x20, 0x35, 0x00, 0x55, 0x55

From Device
0xAA, 0x00, 0xFF, 0x02, 0x20, 0x35, 0x56, 0x55

I see it again here (Host asking about firmware) in the header, just AFTER the packet length byte (4) -
From Host
0xAA, 0x20, 0x01, 0x00, 0x21, 0x55

From Device
0xAA, 0x00, 0xFF, 0x10, 0x20, 0x01 ... M E T R 0 V 0 5 . 0 1 0x00, 0xC4, 0x02, 0x78, 0x55

Yes, I see this.. they change but they match. 0x20 seems consistent.

I am going to try an get some more captures from another device plugged in. Might take some time to find the right place to tap in.

What does this all mean?