Smartmeter, SML code parsing, problems

Hello all!

I have this problem for several months now and since I’m not the big coding insider, it really gives me gray hairs :slight_smile:
Googling and reading postings did not help me to find a solution yet, also weird code behavior puzzles me, so I will write about all of that now and maybe someone has some hints for me.

I have 3 smartmeters here that I want to read out using an ESP32.
The smartmeter shoots out data via an optical port every second (AFAIK this interval is not fix), I successfully get the data into the ESP but here my problems begin.

Currently I use a serial command I manually send to the ESP and a sub to trigger that starts to look for serial data from the smartmeter. Later on I want to put that in a task or using a timer to start the read out and parsing.

I started using a byte array where I store the serial data:

byte Arr_RX1[1000];

To be able to see what the smartmeter sends, I fill the array using this:

Z1_ser.setTimeout(700);
Z1_ser.readBytes(Arr_RX1,500);

for (byte gg = 0; gg < 500; gg++)
{
  char hexChar[3];
  sprintf(hexChar, "%02X", Arr_RX1[gg]);
  Serial.print(hexChar); Serial.print(" ");
}

I want to see hexadecimal values, that’s why I use sprintf.

But here I have a problem: The serial send never ends. It sends a never ending stream of hex values!
gg < 10 for example works. I guess I run into a buffer overload, could it be?
I splitted up the for loop into 5, 100 values at once, but the same neverending send happens in the middle of the 2nd loop.

If I use Serial.write(Arr_RX1,500) it works, but then I have to use a serial monitor with hex view, that’s a little uncomfortable.

How can I serial send an array (or 500 values) to be able to look at the data?

Using google, I found a thread in the German section of this forum and also an example code that I altered a little bit. The original code does not work for me because I don’t know the correct sequences without being able to look at the serial data in hex format (except I use a serial monitor with hex view and get correct data, see the following experience).

This is the thread with the code at the end: https://forum.arduino.cc/index.php?topic=548325.15

I altered this code a little and also edited the start/stop/power sequences (which ich got from a sermon with hex view and the smart meter manual). It looks like this:

defs:
byte Arr_RX1[1000];
const byte startSequence[] = { 0x1B, 0x1B, 0x1B, 0x1B, 0x01, 0x01, 0x01, 0x01 };//start sequence of SML protocol
const byte stopSequence[]  = { 0x1B, 0x1B, 0x1B, 0x1B, 0x18 };//end sequence of SML protocol
const byte powerSequence[] = { 0x07, 0x01, 0x00, 0x01, 0x08, 0x00, 0xFF, 0x64, 0x01, 0x02, 0xE2, 0x01, 0x62, 0x1E, 0x52, 0xFF, 0x56 }; //sequence preceeding the current "pos Wirkenergie, 1.0.8." value (4 Bytes)
int startIndex = 0;//start index for start sequence search
int stopIndex = 0;//start index for stop sequence search
byte power[4]; //array that holds the extracted 4 byte "Wirkleistung" value
int currentpower;
byte inByte;
unsigned int RXx_stage = 0;
unsigned int Arr_RX1_idx = 0;


RXx_stage = 0;
if (RXx_stage == 0)
{
  while (Z1_ser.available())
  {
    inByte = Z1_ser.read();//read serial buffer into array
    if (inByte == startSequence[startIndex])//in case byte in array matches the start sequence at position 0,1,2...
    {
      Arr_RX1[startIndex] = inByte;//set smlMessage element at position 0,1,2 to inByte value
      startIndex++;
      if (startIndex == sizeof(startSequence))//all start sequence values have been identified
      {
        Serial.println("st0 Match found");
        RXx_stage = 1;
        Arr_RX1_idx = startIndex;
        startIndex = 0;
      }
    }
    else {startIndex = 0;}
    if (RXx_stage == 1) {break;} //I added this because else it would read all ser_data and not find the stop sequence (at least it did until I added it)
  }
}

if (RXx_stage == 1)
{
  Serial.println("st1 start");
  while (Z1_ser.available())
  {
    inByte = Z1_ser.read();
    Arr_RX1[Arr_RX1_idx] = inByte;
    Arr_RX1_idx++;
    if (inByte == stopSequence[stopIndex])
    {
      stopIndex++;
      if (stopIndex == sizeof(stopSequence))
      {
        RXx_stage = 2;
        stopIndex = 0;
        // after the stop sequence, ther are sill 3 bytes to come.
        // One for the amount of fillbytes plus two bytes for calculating CRC.
        for (int c = 0 ; c < 3 ; c++)
        {
          Arr_RX1[Arr_RX1_idx++] = Z1_ser.read();
          if (stopIndex == sizeof(stopSequence))
          {
            RXx_stage = 2;
            stopIndex = 0;
            // after the stop sequence passed, there are sill 3 bytes to come.
            // One for the amount of fillbytes plus two bytes for calculating CRC.
            delay(3); // wait for the 3 bytes to be received.
            for (int c = 0 ; c < 3 ; c++)
            {
              Arr_RX1[Arr_RX1_idx++] = Z1_ser.read();
            }
            // smlIndex is at this point one bigger than the amount of stored inBytes because it is incremented evreytime after reading.
            // To store the real smlIndex, we have to substract the last incrementation.
            Arr_RX1_idx--;
          }
        }
        Serial.println("st1 done");
      }
    }
    else {stopIndex = 0;}
  }
}

if (RXx_stage == 2)
{
  Serial.println("st2 start");
  byte temp; //temp variable to store loop search data
  startIndex = 0; //start at position 0 of exctracted SML message
   for(int x = 0; x < sizeof(Arr_RX1); x++) //for as long there are element in the exctracted SML message
  {
    temp = Arr_RX1[x]; //set temp variable to 0,1,2 element in extracted SML message
    if (temp == powerSequence[startIndex]) //compare with power sequence
    {
      startIndex++;
      if (startIndex == sizeof(powerSequence)) //if complete sequence is found
      {
        for(int y = 0; y< 4; y++) //read the next 4 bytes (the actual power value)
        {
          power[y] = Arr_RX1[x+y+1]; //store into power array
        }
        RXx_stage == 3; // go to stage 3
        startIndex = 0;
        Serial.println("st2 Match found");
        currentpower = (power[0] << 24 | power[1] << 16 | power[2] << 8 | power[3]); //merge 4 bytes into single variable to calculate power value
        Serial.print("power:"); Serial.println(currentpower);
      }
    }
    else {startIndex = 0;}
  }
  //currentpower = (power[0] << 24 | power[1] << 16 | power[2] << 8 | power[3]); //merge 4 bytes into single variable to calculate power value
}

if (RXx_stage == 3)
{
  //Serial.print("power:"); Serial.println(currentpower);
  RXx_stage = 0;
}

I put that in a task and let it run all the time.

edit: continued in next posting

Here is what I see in ser_mon:

st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st1 done
st2 start
st2 Match found
power:51248
st0 Match found
st1 start
st0 Match found
st1 start
st1 done
st2 start
st2 Match found
power:51248
st0 Match found
st1 start
st1 done
st2 start
st2 Match found
power:182320
st0 Match found
st1 start
st0 Match found
st1 start
st0 Match found
st1 start
st1 done
st2 start
st2 Match found
power:51248
st0 Match found
st1 start
st0 Match found
st1 start

etc etc. The value for power is always the same 51248, except once it was 182320 ... But it should show something else (increasing value over time)

And here is one transmission I got for the hex values:

1B 1B 1B 1B 01 01 01 01 76 07 00 09 09 00 B1 3F 62 00 62 00 72 63 01 01 74 01 01 07 00 09 02 BE 3B 15 0B 09 01 45 4D 48 00 00 77 3E 1D 01 01 63 BC F5 00 76 07 00 09 01 00 31 40 62 00 62 00 70 63 07 01 77 01 0B 09 01 45 4D 48 00 00 77 3E 1D 07 01 00 62 08 FF FF 72 62 01 65 02 BE 07 9A 7A 77 07 81 81 C7 82 03 FF 01 01 01 01 04 45 4D 48 01 77 07 01 00 00 00 09 FF 01 01 01 01 0B 09 01 45 4D 48 00 00 77 3E 1D 01 77 07 01 00 01 08 00 FF 64 01 02 E2 01 62 1E 52 FF 56 00 00 C8 0A AD 01 77 07 01 00 02 08 00 FF 64 01 02 E2 01 62 1E 52 FF 56 00 00 B2 4E 55 01 77 07 01 00 01 08 01 FF 01 01 62 1E 52 FF 56 00 00 00 00 00 01 77 07 01 00 02 00 01 FF 01 01 60 1E 52 FF 56 00 02 B2 4E 55 01 77 07 01 00 01 08 02 FF 01 01 62 1E 52 FF 56 00 02 C8 2A AD 01 77 07 01 00 00 08 02 FF 01 01 62 1E 52 FF 56 00 00 00 00 00 01 77 07 01 00 10 07 00 FF 01 01 62 1B 52 FF 55 FF FF E2 83 01 77 07 81 81 C7 82 05 FF 01 01 01 01 83 02 E1 1E 72 63 71 72 0E 91 16 A8 DE 85 58 8A D9 2A D2 4B 1F 5E 42 14 7F 71 99 B2 16 55 20 A8 F4 61 77 A7 F5 C1 4F 30 7B 26 A5 50 05 DD D7 84 5C 89 01 01 01 63 55 B4 00 76 07 00 09 09 00 B1 43 62 00 62 00 72 63 00 01 71 01 63 EC B0 00 1B 1B 1B 1B 18 00 39 62 1B 1B 1B 1B 01 01 01 01 76 07 00 09 09 00 B1 45 62 00 62 00 72 63 01 01 76 01 01 07 00 09 02 BE 3B 17 03 09 01 45 4D 48 00 00 77 3E 1D 01 01 63 3B A6 00 74 07 00 09 09 00 B1 46 62 00 62 00 70 63 07 01 77 01 0B 01 01

The data for powerSequence (07 01 00 01 08 00 FF 64 01 02 E2 01 62 1E 52 FF 56) is this one:
07 01 00 01 08 00 FF 64 01 02 E2 01 62 1E 52 FF 56 00 00 C8 0A AD 01
the value should be 00 C8 0A AD I think(!!), I have no info about the length, but it ends before the 01.

That is where I currently am, and whew, that's a bit posting, sorry :slight_smile:
If anyone can help me out with hints or a better solution, that would be really great!

PS:
Actually, I want to read out more than one value from the data above. There are 3 values, one for current active (wattful) power, one for active (wattful) energy fed into and one for active energy sourced from power grid. That means I need to add 2 more sequences and then look for them in the array like it should happen above.

Serial.print(hexCar,HEX); should work :slight_smile:

Are you sure your decoder is working correct?

oh yeah, there was a typo in the code in 1st post, hexCar should have been hexChar like it is in the code
but the send goes into an infinite loop at some point.

The decoder should work correctly, it looks fine on the oscilloscope. (if you meant the hardware)

I still don't see why I have to look for start and stop sequences.
Since the data that's being sent by the smartmeter always contains the needed information in every packet, shouldn't

Z1_ser.setTimeout(800);
Z1_ser.readBytes(Arr_RX1,600);

and then going to "RXx_stage 2" work equally?

If I would have to code this in vb.net, I would just turn it into a big string and then split it using the powerSequence as a string, then take out chars 2 to 9 or something. ;D

I just tried the idea above, I just read 600 bytes with the 800ms timeout and go straight to stage 2.
Today I get this in sermon:

st2 start
st2 start
st2 start
st2 Match found
power:182610
st2 start
st2 Match found
power:51538
st2 start
st2 Match found
power:182610
st2 start
st2 start
st2 start
st2 start
st2 start
st2 start
st2 Match found
power:51538
st2 start
st2 Match found
power:182610
st2 start
st2 Match found
power:51538
st2 start
st2 Match found
power:182610
st2 start
st2 Match found
power:51538
st2 start
st2 start

The 2 values have changed and both appear kinda equally often.

51248 → 51538
182320 → 182610

And another try. I wanted to serial print the array again.
Since I can print like 10 bytes fine and at 100+ I run into problems, I tried to print 30 at once using this:

for (int kk = 0; kk < 20; kk++)
  {
    for (byte tt = 30*kk; tt < 30*(kk+1); tt++)
    {
      char hexChar[3];
      sprintf(hexChar, "%02X", Arr_RX1[tt]);
      Serial.print(hexChar); Serial.print(" ");
    }
  }

It AGAIN goes into an endless loop and I have to reset the ESP32.
I just don’t know what’s wrong :frowning:

The usual suspect with serial: boudrate too low, so you miss essential data.

you mean the baud rate to the PC? that's 115200 :o
I can not change the rate from the smartmeter, but that's 9600 already (some only send at 300)

ok, new test …

I trigger this manually with a serial command (dbg_ctr = 30 on startup):

if (dbg_ctr == 30)
{
  Z1_ser.setTimeout(900);
  Z1_ser.readBytes(Arr_RX1,800);
  dbg_ctr = 0;
}
else
{
  String fck = "";
  for (byte tt = (30*dbg_ctr); tt < 30*(dbg_ctr+1); tt++)
  {
    char hexChar[3];
    sprintf(hexChar, "%02X", Arr_RX1[tt]);
    fck.concat(hexChar); fck.concat(" ");
  }
  fck.concat("-"); fck.concat(dbg_ctr); fck.concat("-");
  Serial.println(fck);
  dbg_ctr++;
  if (dbg_ctr == 20) {dbg_ctr = 30;}
}

After 9 times (1 to read data, 8 to request it) it crashes! No more reaction from the ESP.

Here is what I got 2 times:

1B 1B 13 1B 01 01 01 01 76 07 00 09 09 06 8F 16 62 00 62 00 72 43 01 01 76 01 01 07 00 09 -0-
02 C0 2F B2 0B 09 01 45 4D 48 00 00 77 3C 1D 01 01 63 D6 68 00 76 07 00 09 09 06 8F 17 62 -1-
00 62 00 72 63 07 01 77 01 0B 09 01 45 4D 40 00 00 77 3E 1D 07 01 00 62 0A FF FF 72 62 01 -2-
65 00 C0 0D BD 7A 77 07 81 81 C7 82 03 FF 01 01 01 01 04 45 4D 48 01 77 07 01 00 00 00 09 -3-
FF 01 01 01 01 0B 09 01 45 4D 48 00 00 77 3E 1D 01 77 07 01 00 01 08 00 FF 64 01 02 E2 01 -4-
62 1C 52 FF 56 00 02 C9 57 29 01 77 07 01 00 02 08 00 FF 64 01 02 E2 01 62 1C 52 FF 56 00 -5-
02 B7 D1 51 01 77 07 01 00 01 08 01 FF 01 01 62 1E 52 FF 54 00 00 00 00 00 01 77 07 01 00 -6-
02 08 01 FF 01 01 62 1E 52 FF 56 00 02 B7 D9 52 01 77 07 01 00 01 08 02 FF 01 01 42 1E 52 -7-

1B 1B 1B 1B 01 01 01 01 76 07 00 09 09 06 9A 62 62 00 62 00 72 63 01 01 76 01 01 07 00 09 -0-
02 C0 33 74 0B 09 01 45 4D 48 00 00 77 3E 1D 01 01 43 56 DC 00 76 07 00 09 09 06 98 63 42 -1-
00 62 00 72 63 07 01 77 01 0B 09 01 45 4D 48 00 00 77 3E 1D 07 01 00 42 08 FF FF 72 62 01 -2-
65 02 C0 30 94 7A 77 07 81 81 C7 82 03 FF 01 01 01 01 04 45 4D 48 01 77 07 01 00 00 00 09 -3-
FF 01 01 01 01 0B 09 01 45 4D 48 00 00 77 3E 1D 01 77 07 01 00 01 08 00 FF 64 01 02 E2 01 -4-
62 1E 52 FF 56 00 02 C9 57 29 01 77 07 01 00 00 08 00 FF 64 01 02 E2 01 62 1E 52 FF 56 00 -5-
02 B7 EB E9 01 77 07 01 00 01 08 01 FF 01 01 62 1E 52 FF 56 00 00 00 00 00 01 77 07 01 00 -6-
02 08 01 FF 01 01 62 1E 52 FF 56 00 02 B7 EB E9 01 77 07 01 00 01 00 02 FF 01 01 60 1E 52 -7-

I see that some bytes are not correctly read by the ESP (1B 1B 13 1B should be 1B 1B 1B 1B)
but that’s currently not the main problem :confused:

as a sidenote:
looking at the data (line 4 and 5) 77 07 01 00 01 08 00 FF 64 01 02 E2 01 62 1C 52 FF 56 00 02 C9 57 29 01, the 02 C9 57 29 are 46749481 in decimal. 4674 kWh is what the smartmeter shows on the display :slight_smile:

edit: Just looked into the definition for byte: A byte stores an 8-bit unsigned number, from 0 to 255.
If I want to get the data from the byte array with the size 1000, and I get data from position “30iteration" to "< 30(iteration+1)”, at iteration 7 (and of course 8) it’s bigger than 255.
Is this the reason for the crash and if so, how do I get the rest of data from the array?
I tried using uint before byte, but this also crashed …

edit2: this time “unsigned int” instead of “byte” works to read out the array :o