Help with CRC16-CCITT Reversed checksum and packet

Projects Programming
21

I am going to say my custom packet is invalid for whatever reason. Later, it may become clear.

Here is a valid conversation sample where the 3rd byte rolls over from 0xFF to 0x00,

0xF0 0x05 0xFF 0x00 0x01 0x01 0xD7 0xEB 0xF3
0xF0 0x05 0xFF 0x00 0x00 0xA4 0xA8 0xF3
0xF0 0x05 0x00 0x00 0x01 0x01 0x05 0x2E 0xF3
0xF0 0x05 0x00 0x00 0x00 0x57 0x6E 0xF3

Here, bootup messages are exchanged from power up, starting at 0x00

Time Stamp direction data
5.110334 Slave to Host 0xF0, 0x05, 0x01, 0x01, 0x03, 0x31, 0x30, 0x37, 0xED, 0xF5, 0xF3
7.118354 Host to Slave 0xF4 (framing error), 0xD5, 0xE8, 0xFF
73.270114 Host to Slave 0xFF, 0xFF
464.077116 Host to Slave 0xF0, 0x05, 0x00, 0x02, 0x01, 0x00, 0x34, 0x8A, 0xF3
464.087062 Slave to Host 0xF0, 0x05, 0x01, 0x01, 0x03, 0x31, 0x30, 0x37, 0xED, 0xF5, 0xF3
465.10423 Host to Slave 0xF0, 0x05, 0x01, 0x00, 0x00, 0x8B, 0x34, 0xF3
465.113344 Slave to Host 0xF0, 0x05, 0x02, 0x00, 0x01, 0x01, 0x73, 0x17, 0xF3
466.12937 Host to Slave 0xF0, 0x05, 0x02, 0x00, 0x00, 0xEF, 0xDB, 0xF3
etc...

My first packet should probably be 0xF0, 0x05, 0x00, 0x02, 0x01, 0x00, 0x34, with a checksum of 0x8A, 0xF3

NOT 0xF0,0x05,0x00,0x00,0x00,0x08, 0xF3

More tomorrow!

22

The last two bytes of the message are the 16 bit CRC value, byte swapped. When things aren't working, question the assumptions.

void setup() {
  // put your setup code here, to run once:
Serial.begin(9600);
char a[]={0x05,0x82,0x00,0x00,0x03,0xD7}; //6
char b[]={0x05,0x82,0x00,0x01,0x01,0x1D,0x3A}; //7
char c[]={0x05,0x85,0x05,0x08,0x01,0x04,0x00,0x00,0x00,0x00,0x00,0x00,0x44,0x14}; //14
  Serial.println(CRC16K(a,4),HEX);  //prints D703
  Serial.println(CRC16K(b,5),HEX);  //3A1D
  Serial.println(CRC16K(c,12),HEX); //1444
}
// kermit
uint16_t CRC16K(uint8_t *x, uint8_t len) {
    uint8_t *data = x;
    uint16_t crc=0;

    while (len--) {
        crc ^= *data++;
        for (uint8_t k = 0; k < 8; k++)
            if (crc & 1) crc =(crc >> 1) ^ 0x8408;
            else crc = crc >> 1;
    }

    return crc;
}

void loop() {
  // put your main code here, to run repeatedly:
}
23

Is that only for certain packets? It seems like the case is different for the tests in the earlier replies.

Reply #13,

Testing some more,
char a[]={0xF0,0x05,0x0e,0x00,0x00,0x08};
returns 45E

char a[]={0xF0,0x05,0x15,0x00,0x00,0x08};
returns 7EA

Only the original packet returns a correct checksum..
char a[]={0xF0,0x05,0x23,0x00,0x00,0x08};
returns 82

oh, maybe it is because that last byte in the above tests contains one of the bytes of the checksum.

I will try some tests.

24

OK, that's it!

Last 2 bytes are the checksum (excluding the end of packet marker)!

Yeaaa +999 Karma

So there really isn't much data in most of the packets.

Start marker, [device], sequence number, data, data, checksum, checksum, End of packet marker
I am not sure what I am calling the [device] byte is actually the device, will look at more data. When the device responds from an invalid packet, I think that byte is 0.

ANY idea why they encoded the checksum like they did?

Thank you!

25

excluding the end of packet marker

Both start (F0) and end (F3) of packet symbols are ignored.

ANY idea why they encoded the checksum like they did?

This sort of message format and choice of CRC algorithm goes back around 40 years. KERMIT was a program used to exchange files between computers via telephone modems in the early 1980s, and it used exactly this CRC16.

I should have questioned the "one byte" CRC16 idea earlier (not that you are to blame, you got most of it right).

26

lol

I was around in the 80's. Had 300 baud, 1200 baud, 2400 baud. That was as fast as our Commodores would go. Ran a BBS, the Punter protocol was the dominant file transfer protocol on that platform. I think Kermit is even older. Xmodem! ha. Started out with a tape drive, floppy drives were too expensive. It took about 20 minutes to load Monopoly! Remote control cars sometimes still had wires.

Also had to get up to turn the channel on the TV. UHF, VHF.. :stuck_out_tongue:

Thanks @jremington

27

OK, need to get the 2 bytes into 2 elements of the "SendBuffer[]" variable I have.

Perhaps 2 CRC functions, one to return the hi byte and another to return the low byte.

return crc >> 8; // hi byte
return crc & 0xFF; // low byte

sample - THIS WORKS!! Not pretty but it works! I am in business!

void SendCadence1() {
  Serial.println("Sending Cadence 1...");
  SendBuffer[0] = 0xF0;                //  Begin
  SendBuffer[1] = 0x05;                //  Device?
  SendBuffer[2] = 0x00;                //  Sync byte
  SendBuffer[3] = 0x00;                //  Nothing to say
  SendBuffer[4] = 0x00;                //  Nothing to say
  SendBuffer[5] = CRC16K_low(&SendBuffer[1],4);  //  Calculate Checksum
  SendBuffer[6] = CRC16K_hi(&SendBuffer[1],4);   //  Calculate Checksum
  SendBuffer[7] = 0xF3;  //  End Packet

  for(int i = 0; i < 8; i++) {
    Serial2.write(SendBuffer[i]);
    Serial.print("SendBuffer[");  Serial.print(i);
    Serial.print("] = 0x");  Serial.println(SendBuffer[i], HEX);
  }
}

uint16_t CRC16K_hi(uint8_t *x, uint8_t len) {
    uint8_t *data = x;
    uint16_t crc=0;

    while (len--) {
        crc ^= *data++;
        for (uint8_t k = 0; k < 8; k++)
            if (crc & 1) crc =(crc >> 1) ^ 0x8408;
            else crc = crc >> 1;
    }

    return (crc >> 8);  // High byte
}

uint16_t CRC16K_low(uint8_t *x, uint8_t len) {
    uint8_t *data = x;
    uint16_t crc=0;

    while (len--) {
        crc ^= *data++;
        for (uint8_t k = 0; k < 8; k++)
            if (crc & 1) crc =(crc >> 1) ^ 0x8408;
            else crc = crc >> 1;
    }

    return (crc & 0xFF);  // Low Byte
}
28

Instead of

  SendBuffer[5] = CRC16K_low(&SendBuffer[1],4);  //  Calculate Checksum
  SendBuffer[6] = CRC16K_hi(&SendBuffer[1],4);   //  Calculate Checksum

one can does this (saves time and code space):

uint16_t crc=CRC16K(&SendBuffer[1],4);
  SendBuffer[5] = crc&0xFF;
  SendBuffer[6] = crc>>8;
29

Thanks, it hit me when I took a break.

30

ha, well, here is a bit of a twist..

IF either device winds up having 0xF0 (start) or 0xF3 (end) as part of the checksum, it will replace that byte with 0x1B.

So, need to check if the checksum contains one of those characters, if so add an element

Example,
F0, 05, 16, 00, 00, 1B, 3D, F3

That's easy enough to replicate.

31

No, it seems to want to add an element to the string and re-calc the checksum.

32

Fixed the issue above with the 0xF0, 0xF3, (and I guess 0x1B too) appearing in the checksum and Sequence byte.

@jremington

I found a packet that doesn't seem to work.

char i[]={0xF0,0x05,0x16,0x00,0x00}; //4 0x3D1B
char j[]={0xF0,0x05,0x16,0x00,0x00,0x1B}; //5 0x3D

The original packet captured has -
0xF0,0x05,0x16,0x00,0x00,0x1B,0x5B,0x3D,0xF3

We are getting 0x3D, but NOT 0x5B, just a 1 byte checksum returned.

No 0x5B at all.

0x1B is the byte the devices use to replace 0xF0 and 0xF3.

??

void setup() {
  // put your setup code here, to run once:
Serial.begin(9600);

char i[]={0xF0,0x05,0x16,0x00,0x00}; //4  0x3D1B
char j[]={0xF0,0x05,0x16,0x00,0x00,0x1B}; //5  0x3D

Serial.println(CRC16K(&i[1],4),HEX);
Serial.println(CRC16K(&j[1],5),HEX);


uint16_t CRC16K(uint8_t *x, uint8_t len) {
    uint8_t *data = x;
    uint16_t crc=0;

    while (len--) {
        crc ^= *data++;
        for (uint8_t k = 0; k < 8; k++) 
            if (crc & 1) crc =(crc >> 1) ^ 0x8408;
            else crc = crc >> 1;
    }

    return crc;
}

void loop() {
  // put your main code here, to run repeatedly:
}
33

Sorry, you haven't posted enough information for me to understand the problem. I do understand the need to avoid having F3 in the CRC checksum.

If you have examples of valid messages that don't fit the basic pattern, please post them in their entirety, including the F0 and F3 start and end markers.

34

Sorry, it was late. I edited the above message to include the complete (0xF3) captured trace.
The 0xF3 was not in our CRC16 test program either. That is the end of packet byte. 0xF3 is as constant as 0xF0.

A bit more information..

I am looking through the Boot up messages while the machine boots up and comes online. It takes about 5 minutes to boot. Every 1 second there are PING messages sent from the host to the slave to ensure it is still there. My code sends 7 cadences and then winds up in this NOOP/Ping loop code.
About the 16'th exchange, it farts out on the message above.

As you know, byte 3 increments every time there is an exchange between the two.

So,after the 7 cadences, this is what the bits on the wire look like:

....snip...
0xF0, 0x05, 0x08, 0x00, 0x01, 0x01, 0xDD, 0xCB, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x08, 0x00, 0x00, 0x95, 0xA8, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x09, 0x00, 0x01, 0x01, 0x66, 0xD7, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x09, 0x00, 0x00, 0x40, 0xF2, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0A, 0x00, 0x01, 0x01, 0xAB, 0xF2, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0A, 0x00, 0x00, 0x2D, 0x1D, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0B, 0x00, 0x01, 0x01, 0x10, 0xEE, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0B, 0x00, 0x00, 0xF1, 0x47, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0C, 0x00, 0x01, 0x01, 0x31, 0xB9, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0C, 0x00, 0x00, 0xF4, 0xCB, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0D, 0x00, 0x01, 0x01, 0x8A, 0xA5, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0D, 0x00, 0x00, 0x28, 0x91, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0E, 0x00, 0x01, 0x01, 0x47, 0x80, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0E, 0x00, 0x00, 0x4C, 0x7E, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x0F, 0x00, 0x01, 0x01, 0xFC, 0x9C, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x0F, 0x00, 0x00, 0x90, 0x24, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x10, 0x00, 0x01, 0x01, 0xA4, 0xED, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x10, 0x00, 0x00, 0xC2, 0xEB, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x11, 0x00, 0x01, 0x01, 0x1F, 0xF1, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x11, 0x00, 0x00, 0x1E, 0xB1, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x12, 0x00, 0x01, 0x01, 0xD3, 0xD4, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x12, 0x00, 0x00, 0x7A, 0x5E, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x13, 0x00, 0x01, 0x01, 0x69, 0xC8, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x13, 0x00, 0x00, 0xA6, 0x04, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x14, 0x00, 0x01, 0x01, 0x48, 0x9F, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x14, 0x00, 0x00, 0xA3, 0x88, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x15, 0x00, 0x01, 0x01, 0x1B, 0xB3, 0X83, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x15, 0x00, 0x00, 0x7F, 0xD2, 0xF3 ; // Host to Slave (Ping)
0xF0, 0x05, 0x16, 0x00, 0x01, 0x01, 0x3E, 0xA6, 0xF3 ; // Slave to Host (Pong)
0xF0, 0x05, 0x16, 0x00, 0x00, 0x1B, 0x3D, 0xF3 ; // Host to Slave- should have 0x5B, 0x3D, 0xF3

0xF0, 0x05, 0x16, 0x00, 0x00, 0x1B, 0x5B, 0x3D, 0xF3 ; // Host to Slave- real pack capture

..A length of 9 bytes, whereas all others are a length of 8..

From what I can tell, for some reason, 0x1B is inserted into the packet string (the checksum of 0x05, 0x16, 0x00, 0x00 is 0x3D, 0x1B). This does not match the real packet. ?? The recalculation of the checksum should be 0x3D, 0x5B, but our CRC checker returns only 0x3D.

The slave does not accept this packet, does not increment the sequence byte and responds with -
0xF0, 0x05, 0x16, 0x00, 0x01, 0x02, 0xA5, 0x94, 0xF3.

My code gets its increment of the sequence byte by looking at the slave's sequence byte, since it is the one that increments on a successful packet exchange. Since the slave does not increment this byte on an error, my code does not increment either.

The real packet capture shows 0x17 as the next packet, 0x18, and so on.

Does this help?

35

No, I still don't understand the problem, although a couple of the messages clearly don't fit the pattern already established. You are not explaining yourself clearly.

What host and what slave are you talking about? Why would a host be sending messages that the "slave doesn't accept"?

The real packet capture

What is the "real packet capture"?

Again, I would like to see several examples of valid messages that illustrate the nonstandard pattern. If you know or think you know what information is being sent, it would be helpful to add that.

Note: 0x1B is the "escape character" which, in serial data transfer protocols, is traditionally used to signal that the rules change for the next character in the sequence.

What is your evidence for this assertion?

0x1B is the byte the devices use to replace 0xF0 and 0xF3.