Translating Incoming Binary Data into Values

Hello there,

I'm quite new to arduino and i'm trying to read data from a weather sensor MISOL and I am be able to recieve Binary data and got it to translated to HEX

the weather sensor will send data every 15 sec, before this i was be able to translate the data using python and reading via serial connection (i was comparing the value between the arduino serial monitor and the value I get when I was using python) as shown below

Pyhton Result

The problem is that I want to make the same result on arduino, but I'm not quite sure how to seperate the data into an array or something and translating it into values.

The connection is quite simple im reaading the data from the weather Sensor uisng RS485 module to arduino SoftwareSerial.

this is the code that im using right now for recieving the data (just saying the delay below is quite useless so ignore that bit) :

#include <SoftwareSerial.h>
SoftwareSerial usbcomm(3,2); //RO,DI

int incomingByte = 0;

void setup() {
    
  usbcomm.begin(9600);//Uncomment for Arduino Lenardo
  Serial.begin(9600);
  //while(!usbcomm);//Uncomment for Arduino Lenardo

  pinMode(13, OUTPUT);//Led Connected
  pinMode(8, OUTPUT);//DE/RE Controling pin of RS-485
}

void loop() 
{
  digitalWrite(8,LOW);//DE/RE=LOW Receive Enabled

  for (int i=0; i<21; i++)
  {
  while(!usbcomm.available()); // wait for a character
    incomingByte = usbcomm.read();
    Serial.print(incomingByte,DEC);
  } 
  
  Serial.println("");
  delay(1000);
//digitalWrite(13,HIGH);//Led ON
//delay(2000);
}

I got a refference where someone trying to split the values but I haven't be able to make it work and couldn't quite understand it it is from this StackOverflow forum. I modified the code to my need but couldn't get it to work

#include <SoftwareSerial.h>
SoftwareSerial usbcomm(3,2); //RO,DI

char *CardNumber = " ";
byte CardNumberByte[21];

void setup() 
{
  Serial.begin(9600);
  usbcomm.begin(9600);
  
  pinMode(8, OUTPUT);
}

void loop() 
{
  Misol();
  HEX_Translete();
}

void HEX_Translete()
{
      // Use 'nullptr' or 'NULL' for the second parameter.
  unsigned long number = strtoul( CardNumber, nullptr, 16);

  for(int i=20; i>=0; i--)    // start with lowest byte of number
  {
    CardNumberByte[i] = number & 0xFF;  // or: = byte( number);
    number >>= 8;            // get next byte into position
  }

  for(int i=0; i<21; i++)
  {
    Serial.print("0x");
    Serial.println(CardNumberByte[i], HEX);
  }

  delay(1000);
}

void Misol()
{
  digitalWrite(8,LOW);//DE/RE=LOW Receive Enabled

  for (int i=0; i<21; i++)
  {
  while(!usbcomm.available()); // wait for a character
    CardNumber = char(usbcomm.read(),HEX);
//    Serial.print(incomingByte,DEC);
  } 
}

any Insight or suggestion would be appriciated.

Alarm bells.

Talk us through what you think is happening here.

Start reading up on pointers

Talk us through what you think is happening here.

to be honest im not quite sure my self for this one, what im trying to do is from the RS485 data assign it to CardNumber and translating it like the code i got from stackoverflow but im not sure it works like that

Start reading up on pointers

I'm sorry if im missing something quite new to this forum, what i focusing is on this problem

The problem is that I want to make the same result on arduino, but I'm not quite sure how to seperate the data into an array or something and translating it into values.

CardNumber = char(usbcomm.read(),HEX);

Ok, so Card number is a pointer.
usbcomm.read returns an int, and HEX has the value 16, neither of which is a pointer.

If you set compiler preferences to report all, I'm willing to bet you'd see at least a warning issued here

If you set compiler preferences to report errors, I'm willing to bet you'd see at least a warning issued here

ok i'm not quite sure how pointer works, but i know that the problem is in this area of the code (knowing it doesn;t show any error but won't work) is there a way to assign the value I get to the CardNumber without pointer?

I'm unclear what you're trying to do.

Reading 21 characters (presumably numeric digits) into a buffer and then converting to an unsigned long isn't going to work (unless the number is expressed in binary).

A key point about reading from serial into a buffer, is to ensure there is enough space in the buffer to receive them.

I'm unclear what you're trying to do.

ok so I might forgot some parts for this (mind you this is from my understanding), the sensor will send 21 Binary data in that HEX data there have 42 character and devide it in a set of 2 (like what hex should be an like the one shown in my python code)

I can translate the incoming data to integers, but still what i want to do is just devide the oncoming data into arrays of 21 values, and use the specific values in the array (like array[5] for example) for calculation and get the sensor value like temperature value humidity value etc.

so like i have incoming data of 24ED17E2883F000500000B28818750D8 into

incomingByte []=  {24,ED,17,E2,88,3F,00,05,00,00,0B,28,81,87,50,D8}

I hope it make sense.

This looks like 16 bytes (or 16*8=128 bits) of hexidecimal data. 21 bits of binary data would like:

101000100100101011101

What doesn't seem to be clear is how your sample data would translate to binary.

Taking array[5], for example, it is normally written as 0x3F in hexidecimal, 0b00111111 in binary, or 63 in decimal.

What do you want to get from array[5]? one binary bit value? 8 binary bits? 21 binary bits?

Are you saying that the data comes in as readable hex?
So, say "24" is represented as two ASCII characters with the values 0x32 and 0x34?
Or does the receiver see the hex (binary) value 0x24?

1 Like

This looks like 16 bytes (or 16*8=128 bits) of hexidecimal data. 21 bits of binary data would like: 101000100100101011101

I never tried it in binary but when I translated it to binary it shows like this

What do you want to get from array[5] ? one binary bit value? 8 binary bits? 21 binary bits?

so if you check the Screenshot of the python program that I make it takes an index of a specific HEX fron the data and turned it into integer value, and i'm not sure how to do it here in arduino here is the code snippet from python:

    tx_type = int(data[:2],16)
    security_code = int(data[2:4],16)
    wind_direction = int(data[4:6],16) # wind direction in degrees
    temperature = (int(data[7:10],16)-400)/10 # temperature in celcius
    humidity = int(data[10:12],16) # humidity in percentage
    wind_speed = (int(data[12:14],16)/8)*1.12 # wind speed in m/s
    gust_speed = int(data[14:16],16) * 1.12 # gust_speed in m/s
    accumulation_rainfall = int(data[16:20],16)*0.3 # accumulation rainfall in mm
    uv = int(data[20:24],16) # UV :uW/cm^2
    light = int(data[24:30],16)/10 # Light in Lux

    print(f"Wind Direction  : {wind_direction} degrees")
    print(f"Temperature     : {temperature} celcius")
    print(f"Humidity        : {humidity}%")
    print(f"Wind Speed      : {wind_speed} m/s")
    print(f"Gust Speed      : {gust_speed} m/s")  
    print(f"Rain Rate       : {accumulation_rainfall} mm")
    print(f"UV              : {uv} uW/cm^2")
    print(f"Light           : {light} Lux")
    print()

So, say "24" is represented as two ASCII characters with the values 0x32 and 0x34?
Or does the receiver see the hex (binary) value 0x24?

the reciever should see it as 0x24 as seen here

image

To me Screenshots look mostly like their non-existent alt-text. Sorry.

Ok, in python the int([data[2:4],16) means to use the characters 2&3 from a string, interpreted as a base 16 number, so data is a string of characters:

   # in python: 
   data = "24ED17E2883F000500000B28818750D8"
   tx_type = int(data[:2],16)
   print(tx_type) 
   
   # gives 36

and in C:

   int16_t tx_type;
   char data[] = "24ED17E2883F000500000B28818750D8";
   sscanf(data,"%2x" &tx_type);
   Serial.print(tx_type);

So:

int tx_type,security_code,wind_direction,temperature;
char data[] = "24ED17E2883F000500000B28818750D8";
sscanf(data,"%2x%2x%2x%*1x%3x",&tx_type,&security_code,&wind_direction,
       &temperature);
 temperature_f =(temperature-400)/10;
 Serial.print("TX Type ");
 Serial.println(tx_type);
 Serial.print("Security Code ");
 Serial.println(security_code);
 Serial.print("Wind Direction ");
 Serial.println(wind_direction);
 Serial.print("Temperature ");
 Serial.println(temperature_f);
//...

gives:

TX Type 36
Security Code 237
Wind Direction 23
Temperature 24.00

This is translating incoming ASCII data holding multiple hexadecimal characters into multiple values. sscanf with the %nx format specifier seems like the right answer.

This is translating incoming ASCII data holding multiple hexadecimal characters into multiple values. sscanf with the %nx format specifier seems like the right answer.

ok I've tried what you did almost got it but I think the Data input is not translated to HEX (?) I'm not sure because I disn't get any values except temperature which is abnormal, I think it is directly translated to INT(?) correct me if I'm wrong ( which I think I am).

this is the code right now:

#include <SoftwareSerial.h>
SoftwareSerial usbcomm(3,2); //RO,DI

int incomingByte = 0;
int tx_type,security_code,wind_direction,temperature;


void setup() {
    
  usbcomm.begin(9600);//Uncomment for Arduino Lenardo
  Serial.begin(9600);
  //while(!usbcomm);//Uncomment for Arduino Lenardo

  pinMode(13, OUTPUT);//Led Connected
  pinMode(8, OUTPUT);//DE/RE Controling pin of RS-485
}


void loop() 
{
  digitalWrite(8,LOW);//DE/RE=LOW Receive Enabled

  for (int i=0; i<21; i++)
  {
  while(!usbcomm.available()); // wait for a character
    incomingByte = usbcomm.read(),HEX;
    sscanf(incomingByte,"%2x%2x%2x%*1x%3x",&tx_type,&security_code,&wind_direction,&temperature);
  } 
  int temperature_f =(temperature-400)/10;
     Serial.print("TX Type ");
     Serial.println(tx_type);
     Serial.print("Security Code ");
     Serial.println(security_code);
     Serial.print("Wind Direction ");
     Serial.println(wind_direction);
     Serial.print("Temperature ");
     Serial.println(temperature_f);
}

What's that?
Lose "HEX"

This, because the comma "," operator discards the results of usbcomm.read() and uses the portion after the comma ( HEX happens to equal 16) as the value to assign to the LHS.

Additionally, I'd try a Arduino/Tools/Auto Format to correct the indentation around that for/while.

And also you should be storing the incoming characters in sequential positions in a character buffer, and putting off the parsing until you've read the whole string

char bufferUSB[21+1];

....
   for (int i=0; i<21; i++)
   {
      while(!usbcomm.available())
          ; // wait for a character
      bufferUSB[i] = usbcomm.read();
   } // all the data is read into bufferUSB
   sscanf(incomingByte,"%2x%2x%2x%*1x%3x",&tx_type,&security_code,&wind_direction,&temperature);

...

This, because the comma "," operator discards the results of usbcomm.read() and uses the portion after the comma ( HEX happens to equal 16) as the value to assign to the LHS.

because what I understand is that, I need to translate it into hex then read the hex data from sscanf(incomingByte, "%2x%2x%2x%*1x%3x" that's why I add the hex, but apparently it doesn't work.


but just to make sure we there is no misscom, the data coming is not in HEX, it is just a binary data (from what I read) thats why if I dont translate it into anything it just show gibberish (and preferably to translate the data to hex so I could follow from the python program I made),


also on this part

for (int i=0; i<21; i++)
   {
      while(!usbcomm.available())
          ; // wait for a character
      bufferUSB[i] = usbcomm.read();
   } // all the data is read into bufferUSB
   sscanf(incomingByte,"%2x%2x%2x%*1x%3x",&tx_type,&security_code,&wind_direction,&temperature);

is it suppose to be sscanf(incomingByte,"%2x%2x%2x%*1x%3x", or sscanf(bufferUSB,"%2x%2x%2x%*1x%3x", ? I tried them both but still show the same result

also can I ask how does the sscanf(incomingByte, "%2x%2x%2x%*1x%3x", &tx_type, &security_code, &wind_direction, &temperature); works? i'm not quite understand the %2x etc. (is this the same as modulo, or is it like indexing?) part works

I tried to check the data stored in bufferUSB but it just show the gibberish data


this is the code right now

#include <SoftwareSerial.h>
SoftwareSerial usbcomm(3, 2); //RO,DI

int incomingByte = 0;
int tx_type, security_code, wind_direction, temperature, humidity, wind_speed, gust_speed, rain, uv, light;
char bufferUSB[21 + 1];

void setup() {

  usbcomm.begin(9600);//Uncomment for Arduino Lenardo
  Serial.begin(9600);
  //while(!usbcomm);//Uncomment for Arduino Lenardo

  pinMode(13, OUTPUT);//Led Connected
  pinMode(8, OUTPUT);//DE/RE Controling pin of RS-485
}


void loop()
{
  digitalWrite(8, LOW); //DE/RE=LOW Receive Enabled

  for (int i = 0; i < 21; i++)
  {
    while (!usbcomm.available()); // wait for a character
    bufferUSB[i] = usbcomm.read();
    Serial.print("Incoming Data (for) : ");
    Serial.println(bufferUSB[i], HEX); //just to check the incoming HEX is correct
  }
  Serial.print("Incoming Data : ");
  Serial.println(bufferUSB[0], HEX); //Checking the t_type HEX

  sscanf(bufferUSB, "%2x%2x%2x%*1x%3x%2x%2x%2x%4x%4x%6x", &tx_type, &security_code, &wind_direction, &temperature, &humidity, &wind_speed, &gust_speed, &rain, &uv, &light);

  int temperature_f = (temperature - 400) / 10;
  int wind_speed_f = (wind_speed / 8) * 1.12;
  int gust_speed_f = gust_speed * 1.12;
  int rain_f = rain * 0.3;
  int light_f = light / 10;

  Serial.print("TX Type ");
  Serial.println(tx_type);
  Serial.print("Security Code ");
  Serial.println(security_code);
  Serial.print("Wind Direction ");
  Serial.println(wind_direction);
  Serial.print("Temperature ");
  Serial.println(temperature_f);
  Serial.print("Humidity ");
  Serial.println(humidity);
  Serial.print("Wind Speed ");
  Serial.println(wind_speed_f);
  Serial.print("Gust Speed ");
  Serial.println(gust_speed_f);
  Serial.print("Accumulative Rainfall ");
  Serial.println(rain_f);
  Serial.print("UV ");
  Serial.println(uv);
  Serial.print("Light ");
  Serial.println(light_f);
}

and the result

Incoming Data (for) : 24
Incoming Data (for) : FFFFFFED
Incoming Data (for) : FFFFFFE1
Incoming Data (for) : 62
Incoming Data (for) : FFFFFFAB
Incoming Data (for) : 53
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 5
Incoming Data (for) : 0
Incoming Data (for) : 3
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : FFFFFFAF
Incoming Data (for) : 9
Incoming Data (for) : 1
Incoming Data (for) : FFFFFF87
Incoming Data (for) : FFFFFFA7
Incoming Data (for) : 2F
Incoming Data : 24
TX Type 0
Security Code 0
Wind Direction 0
Temperature -40
Humidity 0
Wind Speed 0
Gust Speed 0
Accumulative Rainfall 0
UV 0
Light 0


and also sorry if it took a while for me to understand this, I never touch data translating in arduino before. and thanks for pointing out about the auto format

Oops. I thought the data is coming in as ASCII text representing hexadecimal data. That is a set of characters like:
"24ED17E2883F000500000B28818750D8"
that is supposed to be broken up into 2-3-4 character chunks and translated into your values:

"24" "ED" "17" "E28" "83" ... F000500000B28818750D8"`

"24" -> 0x24 "ED" -> 0xED, "E28" -> 0xE28 ...

If that's not the case, then sscanf is the wrong tool.

This is a sscanf format string per scanf, fscanf, sscanf, scanf_s, fscanf_s, sscanf_s - cppreference.com that tells how to interpret an array of characters into values. "%2x" means parse 2 characters as if they were hexadecimal The "%*1x" means parse a single char and throw it away.

If you are getting byte data, you could read it all into the bufferUSB and then maybe reassemble it like this:

 // tx_type = int(data[:2],16)
 tx_data = bufferUSB[0]<<8 | bufferUSB[1];
// security_code = int(data[2:4],16)
security_code = bufferUSB[2]<<8| bufferUSB[3];
// wind_direction = int(data[4:6],16) # wind direction in degrees
temp = bufferUSB[7]<<16 | bufferUSB[8] <<8 | bufferUSB[9];
wind_direction = (temp - 400.0))/10;
//...

"%2x" means parse 2 characters as if they were hexadecimal The "%*1x" means parse a single char and throw it away.

then my hunch a quite off but i got the idea thx


If you are getting byte data, you could read it all into the bufferUSB and then maybe reassemble it like this:

I tried using your code, but I think it gets the set of data (?) for now I'm just using the one that you made, and the value seems wrong especially for the temperature

I turned off the HEX translation so and just showing byte data

I think you made a typo here

// wind_direction = int(data[4:6],16) # wind direction in degrees
temp = bufferUSB[7]<<16 | bufferUSB[8] <<8 | bufferUSB[9];
wind_direction = (temp - 400.0))/10;

so I assume it was for temperature not wind direction (?) I might be wrong.

But I changed it to temperature and get -39 C which is immposible here and I tried to mess with the temperature sensor by heating it up but the value doesn't even changed. I also print the temp value by it self and got 5 which is too small.

and I dont think the tx_type and security need to use that syntax knowing the result from bufferUSB[0] alone is already correct the result for tx_type in int should be 36


also what does this opperand do, this << and this |, I check on it that this << is for bit shift but I'm not sure how it works on this when combined with this | opperand for the syntax here

temp = bufferUSB[7]<<16 | bufferUSB[8] <<8 | bufferUSB[9];

I tried to directly use the incoming data from the array just like this

  tx_type = bufferUSB[0];
  security_code = bufferUSB[1];
  wind_direction = bufferUSB[2];

  int temperature_f = (temperature - 400) / 10;
  int wind_speed_f = (wind_speed / 8) * 1.12;
  int gust_speed_f = gust_speed * 1.12;
  int rain_f = rain * 0.3;
  int light_f = light / 10;

  Serial.print("TX Type ");
  Serial.println(tx_type); 
  Serial.print("Security Code ");
  Serial.println(security_code);
  Serial.print("Wind Direction ");
  Serial.println(wind_direction);


it works on the first 3 data array, the tx_type until wind_direction but after that knowing it skip a byte and the next 2 byte is taken for for the temp if i wrote bufferUSB[3] the result doesn't make sense.

24ED17E2883F000500000B28818750D8

"24" (tx_type) , "ED" (security_code), "17" (wind_direction), "E" (skipped), "288" (temperature),..etc. like you decode before

but in this case bufferUSB[3] is "E2"

I changed the char to unsigned char to eliminate any FFFF from the dara result in negatives, but I can only got it until wind_direction.

this is the result

Incoming Data (for) : 24
Incoming Data (for) : ED
Incoming Data (for) : 9C
Incoming Data (for) : 62
Incoming Data (for) : BF
Incoming Data (for) : 47
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 9
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : BD
Incoming Data (for) : DB
Incoming Data (for) : 1
Incoming Data (for) : 87
Incoming Data (for) : 4F
Incoming Data (for) : D7
Incoming Data for HEX tx_type : 24
TX Type 36
Security Code 237
Wind Direction 156
Temperature -40
Humidity 0
Wind Speed 0
Gust Speed 0
Accumulative Rainfall 0
UV 0
Light 0


this is the code right now when using your code, I still need to understand what does it meant.

#include <SoftwareSerial.h>
SoftwareSerial usbcomm(3, 2); //RO,DI

int incomingByte = 0;
int tx_type, security_code, wind_direction, temperature, temp, humidity, wind_speed, gust_speed, rain, uv, light;
unsigned char bufferUSB[21 + 1];

void setup() 
{
  usbcomm.begin(9600);//Uncomment for Arduino Lenardo
  Serial.begin(9600);
  //while(!usbcomm);//Uncomment for Arduino Lenardo

  pinMode(13, OUTPUT);//Led Connected
  pinMode(8, OUTPUT);//DE/RE Controling pin of RS-485
}


void loop()
{
  digitalWrite(8, LOW); //DE/RE=LOW Receive Enabled

  for (int i = 0; i < 21; i++)
  {
    while (!usbcomm.available()); // wait for a character
    bufferUSB[i] = usbcomm.read();
    Serial.print("Incoming Data (for) : ");
    Serial.println(bufferUSB[i],HEX); //just to check the incoming HEX is correct
  }
  
  Serial.print("Incoming Data for HEX tx_type : ");
  Serial.println(bufferUSB[0], HEX); //Checking the t_type HEX

  //sscanf(bufferUSB, "%2x%2x%2x%*1x%3x%2x%2x%2x%4x%4x%6x", &tx_type, &security_code, &wind_direction, &temperature, &humidity, &wind_speed, &gust_speed, &rain, &uv, &light);
  
  tx_type = bufferUSB[0]<<8 | bufferUSB[1];
  security_code = bufferUSB[2]<<8| bufferUSB[3];
  temp = bufferUSB[7]<<16 | bufferUSB[8] <<8 | bufferUSB[9];
  temperature = (temp - 400.0)/10;
//  tx_type = bufferUSB[0];
//  security_code = bufferUSB[1];
//  wind_direction = bufferUSB[2];

//  int temperature_f = (temperature - 400) / 10;
  int wind_speed_f = (wind_speed / 8) * 1.12;
  int gust_speed_f = gust_speed * 1.12;
  int rain_f = rain * 0.3;
  int light_f = light / 10;

  Serial.print("TX Type ");
  Serial.println(tx_type); 
  Serial.print("Security Code ");
  Serial.println(security_code);
  Serial.print("Wind Direction ");
  Serial.println(wind_direction);
  Serial.print("Temperature ");
  Serial.println(temperature);
//  Serial.print("Temperature2 ");
//  Serial.println(temp);
  Serial.print("Humidity ");
  Serial.println(humidity);
  Serial.print("Wind Speed ");
  Serial.println(wind_speed_f);
  Serial.print("Gust Speed ");
  Serial.println(gust_speed_f);
  Serial.print("Accumulative Rainfall ");
  Serial.println(rain_f);
  Serial.print("UV ");
  Serial.println(uv);
  Serial.print("Light ");
  Serial.println(light_f);
}

and the result

Incoming Data (for) : 24
Incoming Data (for) : ED
Incoming Data (for) : 9C
Incoming Data (for) : 62
Incoming Data (for) : BE
Incoming Data (for) : 46
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 9
Incoming Data (for) : 0
Incoming Data (for) : 1
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 0
Incoming Data (for) : 8A
Incoming Data (for) : A7
Incoming Data (for) : 1
Incoming Data (for) : 87
Incoming Data (for) : 66
Incoming Data (for) : EE
Incoming Data for HEX tx_type : 24
TX Type 9453
Security Code -25502
Wind Direction 0
Temperature -39
Humidity 0
Wind Speed 0
Gust Speed 0
Accumulative Rainfall 0
UV 0
Light 0