Need help parsing MQTT data

happytm · March 31, 2025, 8:39pm

I am receiving MQTT message in following format :

["006","106","1","2","3","4","HTM","PASSWORD"]

Now I want to save first 6 items in uint8_t array and last 2 for wifi login SSID and password.

What is the best way to achieve that.

Thanks.

ledsyn · March 31, 2025, 8:46pm

Is this a hidden suggestion for the Arduino project?

pert · March 31, 2025, 9:22pm

I moved your topic to a more appropriate forum category @happytm.

The Community > Suggestions for the Arduino Project category you chose is only to be used to submit suggestions for changes to the Arduino company's software and hardware projects. It is not an appropriate place to publish topics like this that are requesting assistance with your own project.

In the future, when creating a topic please take the time to pick the forum category that best suits the subject of your question. There is an "About the _____ category" topic at the top of each category that explains its purpose.

As you can see from @ledsyn's reply, selecting an inappropriate category can cause confusion or otherwise derail the discussion, and thus make it less likely for you to receive effective assistance.

Thanks in advance for your cooperation.

happytm · March 31, 2025, 9:24pm

Yes this is for Arduino using ESP32.

kenb4 · March 31, 2025, 10:31pm

If you're already using JSON (like with the ArduinoJSON library), it could be deserialized as a JSON array. Otherwise, sscanf is kinda prickly, but would work

void setup() {
  Serial.begin(115200);
  const char *q = R"(["009","106","1","2","3","4","HTM","PASSWORD"])";
  uint8_t n[6];
  const char ssid[6];
  const char password[6];
  auto result = sscanf(q, R"(["%d","%d","%d","%d","%d","%d","%5[^"]","%5[^"]"])",
    n, n+1, n+2, n+3, n+4, n+5, ssid, password
  );
  Serial.println(result);
  Serial.println(n[0]);
  Serial.println(n[1]);
  Serial.println(n[2]);
  Serial.println(n[3]);
  Serial.println(n[4]);
  Serial.println(n[5]);
  Serial.println(ssid);
  Serial.println(password);
}

void loop() {}

assuming that numbers with a leading zero don't actually mean octal (replaced 006 with 009 to see what happens)
for the strings, need to allocate space including the terminating NUL character -- 6 bytes each in this example
which means requesting one byte less for the strings -- 5 bytes here
with a negated scanset, [^"]: anything except a quote

prints

8
9
106
1
2
3
4
HTM
PASSW

Always check the result to make sure it filled in all of them.

happytm · April 1, 2025, 1:27am

Thank you for your solution.

It works standalone but when I try with live mqtt data it does not print anything except last Serial.printf statement with correct real data from mqtt.

The code is below:

  {
  if (payload && strlen(payload)) {
  const char *q = R"(payload)";
  uint8_t n[6];

  auto result = sscanf(q, R"(["%d","%d","%d","%d","%d","%d","%5[^"]","%5[^"]"])",
    n, n+1, n+2, n+3, n+4, n+5, ssid, password
  );
  Serial.println(result);
  Serial.println(n[0]);
  Serial.println(n[1]);
  Serial.println(n[2]);
  Serial.println(n[3]);
  Serial.println(n[4]);
  Serial.println(n[5]);
  Serial.println(ssid);
  Serial.println(password);
  
  Serial.printf("Received message in topic 'command' & message is:- %s\n", payload);      
  }
});

kenb4 · April 1, 2025, 2:38am

The R"()" is a raw string literal, useful for all those double-quotes in the strings.

Creates a string with the value "payload". Since you've already got the payload in a variable, you don't need a new variable at all. Use the existing one as the first argument

auto result = sscanf(payload, // etc

happytm · April 1, 2025, 4:19am

Thank you. It works perfectly.

happytm · April 3, 2025, 10:22pm

When I use following code I get result that shows correct data:

["136","106","2","6","112","32","HTM","password"]
Received message in topic 'command' & message is:- ["136","106","2","6","112","32","HTM","password"]
136
106
2
6
112
32
HTM
password

String ssid, password;
uint8_t receivedCommand[6];
 

mqtt.subscribe("command", [](const char * payload) 
{
  if (payload && strlen(payload)) {
 Serial.println(payload);
 Serial.printf("Received message  in topic 'command' & message is:- %s\n", payload); 
     
  auto result = sscanf(payload, R"(["%u","%u","%u","%u","%u","%u","%5[^"]","%5[^"]"])", 
 &receivedCommand[0], &receivedCommand[1], &receivedCommand[2], 
 &receivedCommand[3], &receivedCommand[4], &receivedCommand[5], &ssid, 
 &password);
  
  Serial.println(receivedCommand[0]);
  Serial.println(receivedCommand[1]);
  Serial.println(receivedCommand[2]);
  Serial.println(receivedCommand[3]);
  Serial.println(receivedCommand[4]);
  Serial.println(receivedCommand[5]);
  Serial.println(ssid.c_str());
  Serial.println(password.c_str()); 
 }
}

kenb4 · April 4, 2025, 2:50am

You can use pointers to those Strings only because on that platform -- ESP32 I'm guessing because you're using Serial.printf -- String implements SSO, Small String Optimization: storing the chars at the start of the String object if they are short enough, no more than 14 bytes. This is a non-portable implementation detail.

In that case, String::c_str returns the address of the String itself. In normal operation, once the string is too long, then it allocates memory on the heap; returning that address with c_str

Directly writing to the memory does not update the String length, so if you use the String in other ways, it will be empty. For these reasons, you should use a char[] instead.

Separately, the format string still has %5[^"] in the format string, which means that should yield no more than 5 characters. But you're getting "password", which is 8. There may be something else with the code you omitted. I could not reproduce your behavior with this sketch

String ssid, password;
uint8_t receivedCommand[6];

void mqtt_recv(const char * payload)
{
  if (payload && strlen(payload)) {
    Serial.println(payload);
    Serial.printf("Received message  in topic 'command' & message is:- %s\n", payload);

    auto result = sscanf(payload, R"(["%u","%u","%u","%u","%u","%u","%5[^"]","%5[^"]"])",
                         &receivedCommand[0], &receivedCommand[1], &receivedCommand[2],
                         &receivedCommand[3], &receivedCommand[4], &receivedCommand[5], &ssid,
                         &password);

    Serial.println(receivedCommand[0]);
    Serial.println(receivedCommand[1]);
    Serial.println(receivedCommand[2]);
    Serial.println(receivedCommand[3]);
    Serial.println(receivedCommand[4]);
    Serial.println(receivedCommand[5]);
    Serial.println(ssid.c_str());
    Serial.println(password.c_str());
    Serial.println(password.length());  // added this
  }
}
void setup() {
  Serial.begin(115200);
  mqtt_recv(R"(["136","106","2","6","112","32","HTM","password"])");
  String x;
  const char *p = x.c_str();
  do {
    x += static_cast<char>('a' + x.length());
    Serial.printf("%2d %x %s\n", x.length(), x.c_str(), x.c_str());
  } while (x.c_str() == p);
  Serial.printf("sizeof(String): %d\n", sizeof(String));
}

void loop() {}

I got instead

["136","106","2","6","112","32","HTM","password"]
Received message  in topic 'command' & message is:- ["136","106","2","6","112","32","HTM","password"]
136
106
2
6
112
32
HTM
passw
0
 1 3ffb223c a
 2 3ffb223c ab
 3 3ffb223c abc
 4 3ffb223c abcd
 5 3ffb223c abcde
 6 3ffb223c abcdef
 7 3ffb223c abcdefg
 8 3ffb223c abcdefgh
 9 3ffb223c abcdefghi
10 3ffb223c abcdefghij
11 3ffb223c abcdefghijk
12 3ffb223c abcdefghijkl
13 3ffb223c abcdefghijklm
14 3ffb223c abcdefghijklmn
15 3ffb8998 abcdefghijklmno
sizeof(String): 16

happytm · April 4, 2025, 7:10pm

The code is part of larger project so I hesitated to post full code but link below has full code:

github.com/happytm/BatteryNode

examples/SimpleGateway/batteryNodeServer.ino

master


#define FIRSTTIME  false  // Define true if setting up Gateway for first time.

#include <WiFi.h>
#include <esp_wifi.h>
#include <EEPROM.h>
#include "FS.h"
#include <SPIFFS.h>
#include <PicoMQTT.h>
#include <PicoWebsocket.h>
#include <WebServer.h>
#include <motionDetector.h> // Manually install this library from https://github.com/happytm/MotionDetector/tree/main/motionDetector
#include "time.h"

WebServer server(80);WiFiServer websocket_underlying_server(81);WiFiServer tcp_server(1883);
PicoWebsocket::Server<::WiFiServer> websocket_server(websocket_underlying_server);PicoMQTT::Server mqtt(tcp_server, websocket_server);

struct tm timeinfo;
#define MY_TZ "EST5EDT,M3.2.0,M11.1.0" //(New York) https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv

This file has been truncated. show original

The code in question starts at line 379.

Thanks.

kenb4 · April 5, 2025, 4:01am

Ah: instead of the format string allowing 5 bytes each, that code allows 20

And the relevant portion of the variable declarations is

As shown previously, a String object occupies 16 bytes. The way the code works, the layout with SSO

  struct _sso {
    char buff[SSOSIZE];
    unsigned char len   : 7;  // Ensure only one byte is allocated by GCC for the bitfields
    unsigned char isSSO : 1;
  } __attribute__((packed));  // Ensure that GCC doesn't expand the flag byte to a 32-bit word for alignment issues

has the char buffer, which must include the terminating NUL, followed by a bit field: 7 bits for the length (more than enough) and a single bit indicating that it is SSO mode. It's actually part of a union

  union {
    struct _ptr ptr;
    struct _sso sso;
  };

which when not in SSO mode, is

  struct _ptr {
    char *buff;
    uint32_t cap;
    uint32_t len;
  };

SSOSIZE is defined according to that latter struct

  // This allows strings up up to 11 (10 + \0 termination) without any extra space.
  enum {
    SSOSIZE = sizeof(struct _ptr) + 4 - 1
  };

The comment is incorrect (it's old and has not been updated as the code has been ported, if it was ever correct)

struct _ptr is 12 bytes
SSOSIZE is 15
SSO string length is max 14, since it also has to store the terminating NUL, as shown previously

So TL;DR: you can't store 20 bytes there. The memory layout for those String variables is "backwards", with the earlier variables later in memory; with 12-, 13-, and 14-byte strings it would be

           0123456789abcdef  z=NUL  b=bitfield
           ----------------
graphData  123456789   14zb
password   123456789  13z b
ssid       123456789 12z  b

The test reads HTM and password, which results in

           0123456789abcdef  z=NUL  b=bitfield
           ----------------
graphData  123456789   14zb
password   passwordz  13z b
ssid       HTMz56789 12z  b

So luckily, the code still works if you stick with c_str(). (The length() has not been updated.) But if you read a long password like TWENTY-BYTE-PASSWORD

           0123456789abcdef  z=NUL  b=bitfield
           ----------------
graphData  123456789   14zb
password   TWENTY-BYTE-PASS
ssid       WORDz6789 12z  b

This is a classic "why C is an unsafe language" buffer overflow.

ssid.c_str() points to NUL-terminated WORD, which has overwritten HTM
however the length in the bitfield has (again) not been updated, so it still has 12
meanwhile the bitfield for password has been obliterated
- the high bit of the last byte has been cleared, making the isSSO flag false

Now the memory layout is that of struct _ptr. If you print password.c_str() and password.length() in hex (the capacity is not immediately accessible)

4E455754
2D455459

Those numbers should look familiar (when read backwards, or little-endian)

$ echo -n "TWENTY-BYTE-PASS" | hexdump -C
00000000  54 57 45 4e 54 59 2d 42  59 54 45 2d 50 41 53 53  |TWENTY-BYTE-PASS|
00000010

That means you definitely not try to use password as a String, since the pointer and length are invalid. Here's a demo sketch

String ssid, password, graphData, Hour, Minute;

void setup() {
  Serial.begin(115200);
  ssid      = "123456789 12";
  password  = "123456789  13";
  graphData = "123456789   14";

  Serial.println(reinterpret_cast<uintptr_t>(&ssid), HEX);
  Serial.println(reinterpret_cast<uintptr_t>(&password), HEX);
  Serial.println(reinterpret_cast<uintptr_t>(&graphData), HEX);

  sscanf("HTM TWENTY-BYTE-PASSWORD", "%20s %20s", &ssid, &password);

  Serial.println(ssid.c_str());
  Serial.println(ssid.length());
  Serial.println(reinterpret_cast<uintptr_t>(password.c_str()), HEX);
  Serial.println(password.length(), HEX);
  // don't uncomment this
  // Serial.println(password);
}

void loop() {}

Therefore

Use an appropriately-sized plain char array

```cpp
char ssid[21], password[21];
String graphData, Hour, Minute;

with those array names/pointers as arguments to sscanf: ssid, password);

happytm · April 5, 2025, 4:19am

Thank you for explaining in all these detail.

Topic		Replies	Views
Json Parsing Issue Programming	19	216	November 22, 2025
Help needed with sending serial data Programming	34	348	June 17, 2026
MQTT Werte wieder richtig zuordnen Deutsch	55	885	March 22, 2025
Storing Pointer Address Text into a Variable Programming	17	333	November 27, 2025
Parsing MQTT Payload Data Programming	21	5275	June 16, 2023

Need help parsing MQTT data

Therefore

Related topics