Using String variable to store and send binary hex data

I have this function mqtt.publish(topic, message) where the function parameters are Arduino String objects. It uses the MQTT protocol to send data out. Usually, the parameters are in text String. However, in my case, the data is in binary hex.

Here is the relevant code;

MqttClient mqtt("192.168.1.71", 1883, onMessageReceived);
char buffer[100];
//buffer filled with binary data terminated with null character at this point
mqtt_msg = String(buffer);
String topic = "XXX";
mqtt.publish(topic, mqtt_msg );

I will terminate the binary data in buffer with a null character 0x00. The problem comes when 0x00 happened to be part of the binary data. How can this problem be solved?

Don’t terminate binary data with a null character. null characters are string terminators, not binary data terminators.

Instead, store the length of the binary data separately.

You will need to convert the binary data to ASCII (probably as HEX) representation before storing it in a String, if indeed you need to store it in a String.

for example,

binary data: 0xFF, 0x00, 0xCC (3 bytes)

needs to be converted to the character array

“FF00CC\0” (7 bytes)

For now, you could create an empty String object, then simply append the next binary byte (as a HEX string) in a for loop.

e.g. (untested):

String myString;

for (int i=0, i < numBytes; i++)
{
    myString += String(buffer[i], HEX);
}

but that’s really slow.

You might go look up the union structure in C. It looks like this

union chameleon {

String str; byte bin[20]; }

Then you define a variable of type chameleon, say

chameleon buffer;

You can load bytes (i.e., your binary data) in this with the notation buffer.bin = myByte; When you've loaded all the bytes you want, you can do Serial.write(buffer.str); If you haven't filled the whole string, pad it. This is a great technique, I've used it to transform floats to a data stream of bytes, and recover them as floats on the other end.

jrdoner: You might go look up the union structure in C. It looks like this

union chameleon {

String str; byte bin[20]; }

Then you define a variable of type chameleon, say

chameleon buffer;

You can load bytes (i.e., your binary data) in this with the notation buffer.bin = myByte; When you've loaded all the bytes you want, you can do Serial.write(buffer.str); If you haven't filled the whole string, pad it. This is a great technique, I've used it to transform floats to a data stream of bytes, and recover them as floats on the other end. [/quote] I do not have an Arduino here, but this seems to be all kinds of wrong. A String Object is not a character array.

arduinodlb:
Don’t terminate binary data with a null character. null characters are string terminators, not binary data terminators.

Instead, store the length of the binary data separately.

You will need to convert the binary data to ASCII (probably as HEX) representation before storing it in a String, if indeed you need to store it in a String.

for example,

binary data: 0xFF, 0x00, 0xCC (3 bytes)

needs to be converted to the character array

“FF00CC\0” (7 bytes)

For now, you could create an empty String object, then simply append the next binary byte (as a HEX string) in a for loop.

e.g. (untested):

String myString;

for (int i=0, i < numBytes; i++)
{
    myString += String(buffer[i], HEX);
}




but that's really slow.

There is a problem with your code.

Suppose binary data: 0xFF, 0x00, 0x01, 0xCC

myString will contain FF01CC, not FF0001CC

lightaiyee: There is a problem with your code.

Suppose binary data: 0xFF, 0x00, 0x01, 0xCC

myString will contain FF01CC, not FF0001CC

No it won't. It will contain FF0001CC.

edit: You're correct. My mistake. see below.

arduinodlb:
No it won’t. It will contain FF0001CC.

Nope.

byte toConvert[] = { 0xff, 0, 1, 0xCC, };

void setup() {
  Serial.begin(115200);
  String result;
  for (byte idx = 0; idx < sizeof(toConvert); idx++) {
    result += String(toConvert[idx], HEX);
  }
  Serial.print(F("1. result = "));
  Serial.println(result);
  result = "";
  for (byte idx = 0; idx < sizeof(toConvert); idx++) {
    if (toConvert[idx] < 16) {
      result += "0";
    }
    result += String(toConvert[idx], HEX);
  }
  Serial.print(F("2. result = "));
  Serial.println(result);
}
void loop() {}

Output

1. result = ff01cc
2. result = ff0001cc
1 Like

Ahhh. I see.

String((byte)1, HEX); gives "1" not "01".

My mistake. I have never used it. I thought the OP was saying it would skip the 0x00. I also thought that it would generate 2 digits for the byte.

Seeing is believing, and testing code suggestions is a must, especially when one did not use the technique before. ;)

Leading zeros are handled by the conversion functions disregarding the base, humans tend to think different.

Whandall: Seeing is believing, and testing code suggestions is a must, especially when one did not use the technique before. ;)

Leading zeros are handled by the conversion functions disregarding the base, humans tend to think different.

"a must". Hmmm. We disagree on this point. It's all about allocation of time and resources. Plus, I don't have an arduino here, so I can either not help at all, or occasionally be wrong. I choose the latter.

"think different" -> "think differently". Apple is a special case, since they meant both "think differently" and "think -> 'different'". When posting smack-downs, correct grammar is a must. :)

I would prefer using char arrays (and uppercase hexdigits) anyway, like this:

byte toConvert[] = { 0xff, 0, 1, 0xCC, };
char charBuff[9];

void setup() {
  Serial.begin(115200);
  shex(charBuff, toConvert, sizeof(toConvert));
  Serial.print(F("to char[] result = "));
  Serial.println(charBuff);
}
void shex(char* store, byte* from, byte len) {
  while (len--) {
    *store++ = nibble(*from>>4);
    *store++ = nibble(*from++);
  }
  *store = 0;
}
byte nibble(byte val) {
  val &= 0xF;
  return val+(val<10 ? '0' : 'A'-10);
}
void loop() {}
to char[] result = FF0001CC

There are many ways to skin a cat. As long as the OP finds a solution that works for him/her. The world is grey/gray. :)

Thank you, Whandall, for the correct code.

Thank you, arduinodlb, for suggesting a good approach to solving the problem.

The only thing I can thank both of you is to give you Karma points :)

Hi Whandall,

May I ask why do you prefer your second solution? I find the first one more readable.

I avoid using the String class (especially for byte concatenation) wherever possible, because it can lead to memory fragmentation and instabilities. Using char[] gives me much better control over the used memory.

And hexdigits in uppercase are just a matter of taste. 0xfeed vs 0xFEED, i prefer the latter.