CRC16 calculation of P1 DSMR telegram on ESP32

Hi all

I'm building a DSMR P1 digital meter emulator using an ESP32 programmed through Arduino IDE, but I'm having trouble calculating a valid CRC16 checksum. Perhaps an extra set of eyes might point me in the right direction.

A DSMR P1 meter outputs an ASCII message (the meter "telegram") containing all measured data points. An example of this telegram can be found here. Every line ends with a carriage return (CR), followed by a line feed (LF) (see page 15 here).

The final two bytes of the telegram are a CRC16 value calculated over the preceding characters in the telegram (from “/” to and including “!”), using the polynomial x16+x15+x2+1, no XOR in, no XOR out, computed with least significant bit first. This is also called CRC16-IBM.

I've written the following code which, as far as I can establish, should calculate the correct CRC16. For the example telegram, this should be EF2F, but when testing the code on the ESP32 gives me something else (E47C).

uint16_t crc16(const uint8_t *data) {
    uint16_t crc = 0x0000; // initial value
    while (*data) {
        crc ^= (uint8_t)(*data++);  // xor byte into least sig. byte of crc
        for (int i = 0; i < 8; i++) {
            if (crc & 1)
                crc = (crc >> 1) ^ 0xA001;
            else
                crc >>= 1;
        }
    }
    return crc;
}

void setup() {
  Serial.begin(115200); 
  delay(2000);
  const uint8_t data[] = "/ISk5\\2MT382-1000\r\n" //escape the \ in the telegram here
  "\r\n"
  "1-3:0.2.8(50)\r\n"
  "0-0:1.0.0(101209113020W)\r\n"
  "0-0:96.1.1(4B384547303034303436333935353037)\r\n"
  "1-0:1.8.1(123456.789*kWh)\r\n"
  "1-0:1.8.2(123456.789*kWh)\r\n"
  "1-0:2.8.1(123456.789*kWh)\r\n"
  "1-0:2.8.2(123456.789*kWh)\r\n"
  "0-0:96.14.0(0002)\r\n"
  "1-0:1.7.0(01.193*kW)\r\n"
  "1-0:2.7.0(00.000*kW)\r\n"
  "0-0:96.7.21(00004)\r\n"
  "0-0:96.7.9(00002)\r\n"
  "1-0:99.97.0(2)(0-0:96.7.19)(101208152415W)(0000000240*s)(101208151004W)(0000000301*s)\r\n"
  "1-0:32.32.0(00002)\r\n"
  "1-0:52.32.0(00001)\r\n"
  "1-0:72.32.0(00000)\r\n"
  "1-0:32.36.0(00000)\r\n"
  "1-0:52.36.0(00003)\r\n"
  "1-0:72.36.0(00000)\r\n"
  "0-0:96.13.0(303132333435363738393A3B3C3D3E3F303132333435363738393A3B3C3D3E3F303132333435363738393A3B3C3D3E3F303132333435363738393A3B3C3D3E3F303132333435363738393A3B3C3D3E3F)\r\n"
  "1-0:32.7.0(220.1*V)\r\n"
  "1-0:52.7.0(220.2*V)\r\n"
  "1-0:72.7.0(220.3*V)\r\n"
  "1-0:31.7.0(001*A)\r\n"
  "1-0:51.7.0(002*A)\r\n"
  "1-0:71.7.0(003*A)\r\n"
  "1-0:21.7.0(01.111*kW)\r\n"
  "1-0:41.7.0(02.222*kW)\r\n"
  "1-0:61.7.0(03.333*kW)\r\n"
  "1-0:22.7.0(04.444*kW)\r\n"
  "1-0:42.7.0(05.555*kW)\r\n"
  "1-0:62.7.0(06.666*kW)\r\n"
  "0-1:24.1.0(003)\r\n"
  "0-1:96.1.0(3232323241424344313233343536373839)\r\n"
  "0-1:24.2.1(101209112500W)(12785.123*m3)\r\n"
  "!"; //should give EF2F as CRC16

  uint16_t computedCRC = crc16(data);
  Serial.print("Computed CRC16: 0x");
  Serial.println(computedCRC, HEX);
}

void loop() {

}

I've tried various things:

  • Adding /0 to the end of the telegram
  • Not escaping/triple escaping the \ character
  • Using char in stead of uint8_t for the telegram and CRC calculation
  • Using a String object, casting it to char using toCharArray
  • Memcpy the char array from previous point to a uint8_t array to pass it to the CRC function

I believe the root of the issue lies in the use of the escape character \ (backslash), especially concerning the CR and LF. If I use this code to calculate the CRC16-IBM of a (shorter) ASCII string containing no \n or \r and with the \ double escaped, the returned value matches what I get from online calculators. But when inserting a \n or \r, the value is off.

Perhaps this might also have something to do with char being unsigned on ESP32 but signed on AVR Arduino?

Tl,dr: how to calculate the correct CRC16-IBM from the example ASCII string?

There are several on line calculators that test CRC algorithms. Test yours against theirs, using the same ASCII test message.

One example: https://crccalc.com/

But when inserting a \n or \r, the value is off.

Most ASCII CRC calculations ignore non-printing and control characters, so leave those out. Forget about Strings, use unsigned char data type.

I have same issue.
If I compare the calculated CRC from the ESP32 with the one generated here Online Checksum Calculator(CRC-16- IBM)
both matches as long as I do not include a backslash "" in the data package. not only \r , \n but also the one in the ID of the DSRM can't be present.
It is not the tool as when I compare a generated telegam as is inlcuding the "" both matches.
@ epyonbe did you succeed?

Found the issue.

Arduino acts different when a single "" is used.
When I replace "" with "\" , both matches.
Any tips to fix this?

how I see it: first do a find "" and replace with "\" prior to generate the CRC

I don't know for your specific case, but as long as you properly escape the control characters you should be fine.

My problem was that the CRC's in the example telegrams aren't correct. The code to calculate the CRC16's was.

epyonbe, did you get this sorted in the end? I also would like to use the CRC in the message from a power meter using the P1 HAN port. I can read the messages OK but I fail the CRC calculations just as you did. I have tried all sorts of online calculators but they all produce another result than I need. Your description regarding polynomial, XOR etc. is the same as in my case. I have the Aidon 6534 meter and I'm located in Sweden (not that it matters).

I now discovered that the Scadacore online calculator produces the correct results for me too (Reversed 0xA001, little endian). I note that each newline is treated (in the hex translated box) as 0D0A which was one of my questions. I will probably be using an RPi for this but the code should be very similar I guess.

If you or anyone is willing to share some code or tricks to get this working, I'd be very grateful!

@fb35523

Can you post an example string and the CRC it should generate?

Sure, thanks for replying! Here is one example (note the extra empty line after the first):

/ADN9 6534

0-0:1.0.0(240108163800W)
1-0:1.8.0(00010116.516kWh)
1-0:2.8.0(00000000.000
kWh)
1-0:3.8.0(00001759.737kVArh)
1-0:4.8.0(00001260.519
kVArh)
1-0:1.7.0(0004.232kW)
1-0:2.7.0(0000.000
kW)
1-0:3.7.0(0001.304kVAr)
1-0:4.7.0(0000.000
kVAr)
1-0:21.7.0(0001.461kW)
1-0:22.7.0(0000.000
kW)
1-0:41.7.0(0001.242kW)
1-0:42.7.0(0000.000
kW)
1-0:61.7.0(0001.532kW)
1-0:62.7.0(0000.000
kW)
1-0:23.7.0(0000.535kVAr)
1-0:24.7.0(0000.000
kVAr)
1-0:43.7.0(0000.489kVAr)
1-0:44.7.0(0000.000
kVAr)
1-0:63.7.0(0000.276kVAr)
1-0:64.7.0(0000.000
kVAr)
1-0:32.7.0(222.9V)
1-0:52.7.0(225.2
V)
1-0:72.7.0(223.5V)
1-0:31.7.0(007.0
A)
1-0:51.7.0(005.9A)
1-0:71.7.0(006.9
A)
!

CRC from meter is: 1B7D
The CRC is printed on the last line, just after the ! so the actual last line is this:

!1B7D

The manual for the meter says the initial / and the ! on the last line should be included in the calculations. I can also verify that the CRC 1B7D is the same with the Scadacore calculator gives.

Thanks!

If you want another one to play with, here it is. I posted another one just minutes ago in this post, but that one was faulty so I edited the post. This one is correct.

/ADN9 6534

0-0:1.0.0(240108164940W)
1-0:1.8.0(00010117.368kWh)
1-0:2.8.0(00000000.000
kWh)
1-0:3.8.0(00001759.974kVArh)
1-0:4.8.0(00001260.519
kVArh)
1-0:1.7.0(0004.251kW)
1-0:2.7.0(0000.000
kW)
1-0:3.7.0(0001.279kVAr)
1-0:4.7.0(0000.000
kVAr)
1-0:21.7.0(0001.462kW)
1-0:22.7.0(0000.000
kW)
1-0:41.7.0(0001.263kW)
1-0:42.7.0(0000.000
kW)
1-0:61.7.0(0001.523kW)
1-0:62.7.0(0000.000
kW)
1-0:23.7.0(0000.529kVAr)
1-0:24.7.0(0000.000
kVAr)
1-0:43.7.0(0000.500kVAr)
1-0:44.7.0(0000.000
kVAr)
1-0:63.7.0(0000.269kVAr)
1-0:64.7.0(0000.000
kVAr)
1-0:32.7.0(221.2V)
1-0:52.7.0(224.3
V)
1-0:72.7.0(223.3V)
1-0:31.7.0(007.0
A)
1-0:51.7.0(006.0A)
1-0:71.7.0(006.9
A)
!759E

Hi. I did figure it out, yes. Turns out the CRC values in the official P1 documentation, and examples based on them, are wrong. So there was nothing wrong with my code after all. Used it on telegrams coming from my meter (when I got one) and it worked.

Ok, now I got the hang of it. Mind you, I'm new to Python and can hardly be called a programmer at all, but here is some sample code that gives me what I need. As my Python 3 is only version 3.5 (I know, upgrade) on the RPi I have available for this, the code cannot use some features like f-strings that I came to stumble upon. I will upgrade soon...

The string "message" contains the telegram from the meter, excluding the CRC, so from / to !, inclusive. I use crcmod for this, so install and import that first. The first line creates the crc function on the fly with the parameters I feed into it (totally new feature to me, but handy!). rev=1 is default, but I include it anyway.

      crc16_P1=crcmod.mkCrcFun(0x18005, initCrc=0, rev=1, xorOut=0x0000) # Create function
      mess2 = bytes(message, 'ascii') # Convert the telegram into byte string for the crc function
      crc16b=crc16_P1(mess2) # Runs the actual crc calculation
      print(hex(crc16b)) # Result is like 0xeec5
      crc16c='%0*X' % (4,crc16b)

The last line gives me a crc string that is formatted as the one in the telegram, like EEC5, also making sure leading zeros are added as needed to mimic the telegram crc. Now I just need to check each telegram so the crc is correct and decide what to do if it is not.

Thanks everyone, I hope this helps someone else too! (I know this is an Arduino forum, but help on the matter is scarce and only the best (=Arduino experts) seem to have even a clue :wink: )

You're not the only one that made the incorrect assumption that an example in the specification is the golden reference :smiley: I figured it out based on real meter data using a CRC algorithm by @matthijskooijman: arduino-dsmr/src/dsmr/crc16.h at master · matthijskooijman/arduino-dsmr · GitHub

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.