Combining data in as few bytes as possible

Hello,

Sorry for writing in "bad' English, I'm living in the Flemish speaking part of Belgium.

I'm looking for an algorithm to compress data in as less bytes as possible to send them by LoRaWAN.
Of course, I also have to reconstruct the original data.

The data I have to send consists of:

3 temperatures ranging from -25.0 °C to 99.0 °C (with .1 decimal precision)
1 temperature ranging from -25 °C to 999 °C (with no decimals)
4 voltages ranging from 0.0 to 25.0 volt (with .1 decimal precision)
8 switch states (can be combined in 1 byte)

The code to retrieve these values is already written and uses the following variables:

float temp1
float temp2
float temp3
int16_t temp4

float volt1
float volt2
float volt3
float volt4

int8_t switch

What is the best practice to send these values in as less bytes as possible (I think LoRaWAN payload is max 12 bytes)

I was thinking:

  • multiplying the voltage floats by 10 to get rid of the decimal part, so they fit in an int8_t
  • adding 100 to the temp floats to get rid of the negative sign, then multiplying by 100 to fit in an int16_t (2 bytes)

Does anyone have better ideas?

Thanks in advance,
Michel

3x 2 bytes

1x 2 bytes

4x 1 byte

1 byte

Total: 13 bytes

I don't think it is. Can you post a link where that is written?

Well, your English is a whole lot better than my Flemish!

But, FYI, the English would say, "as few bytes as possible"

Definitely.

An int16_t is signed - so there's no need to get rid of the sign.

Is it important to you to keep to standard C/C++ types?

You could save more space by using fewer bits; eg,

Your -25.0 °C to 99.0 °C becomes -250 to 990 which would fit into 11 bits...

It does depend on the Data Rate (DR):

https://lora-developers.semtech.com/documentation/tech-papers-and-guides/the-book/packet-size-considerations/

1 Like

Thanks @awneil I learned something new.

if you want to use the same packet format in the U.S. and the EU, and you want to support DR0 in both regions, the maximum payload length for any packet would be 11 bytes (the U.S. limit for DR0)

I guess I didn't ever need to know that because I'm in Europe (geographically but unfortunately not politically :frowning_face: ).

Hello PaulRB

The Fair Use Policy:

[[Fair Use Policy explained - End Devices (Nodes) - The Things Network]]

I only need to send this data twice a day, except when one of the values goes outside the predefined alarm limits then I have to transmit it also

you are thinking of Sigfox (max uplink 12bytes max downlink 8bytes)
I think LoRaWAN can be up to 250 bytes
however, it is recommended to send as few bytes as possible so don't transmit ASCII text (I have heard of people attempting to transmit JSON!)
if possible encode your float values into 8bit bytes (unsigned 0 to 255 signed -128 to 127) or 16bit ints (unsigned 65535 signed -32768 to 32767) - at the LoRaWAN server you convert back to floats using the payload formatter, e.g.

function decodeUplink(input) {
  var data = {};
 data.Cycles= input.bytes[0]; 
data.unitON= input.bytes[1]; 
data.unitOFF= input.bytes[2]; 
data.data1= input.bytes[3]*10; 
data.temp= input.bytes[4]*10; 
data.pressure= input.bytes[5]*10; 
data.humidity= input.bytes[6]*10; 
  return {
    data: data,
    warnings: warnings
  };
}

Do you really need that level of precision?

See my earlier reply - it does depend on the Data Rate (DR).

When you're really pushing the limits of range, it can be a s low as 11 bytes...

A total of 250+990 = 1240 possible values. 11 bits

3x 11 = 33 bits

1240 values, 11 bits

250 possible values, 8 bits

4x 8 = 32 bits

8 bits

Total: 84 bits = 11 bytes

It could fit in theory!

1 Like

you are thinking of Sigfox (max uplink 12bytes max downlink 8bytes)
I think LoRaWAN can be up to 250 bytes

I will use "The Things Stack Community Edition" as backend.

Do you really need that level of precision?

Unfortunately yes, voltages are measured at different types of batteries, standard lead acid but also LiFe.
And their condition can be checked by analyzing their voltage precisely.

Temperatures, are less important and could be without decimal part, but it would be nice to have them with decimal precision 0.1 °C

how often are you transmitting?
in a recent project I was transmitting 12 bytes every 10 minutes which was well within the recommended limits - have a look at LoRaWAN fair use policy

uint8_t msg[11];
uint8_t bitPos;
uint8_t bytePos;

void addDataBits(uint16_t data, uint8_t bits) {
  for (uint8_t b=0; b<bits; b++) {
    bitWrite(msg[bytePos], bitPos, bitRead(data, b));
    if (++bitPos == 8) {
      bitPos = 0;
      bytePos++;
    }
  }
}

...

  bitPos = 0;
  bytePos = 0;
  addDataBits((temp1 + 25.0) * 10.0, 11);
  ...
  addDataBits(temp4 + 25.0, 11);
  addDataBits(volt1 * 10.0, 8);
  ...
  addDataBits(switch, 8);

Of course it would!

Engineering is always about compromise...

Perhaps you could have 2 (or more) packet formats: sometimes you send just "coarse" data - which takes fewer bits - and sometimes you send the full precision.

Remember that sending more bits also consumes more power...

packet:

-250 to 990 = 1241 --> 11 bits x 3 =  33                       
-25 to 999 = 1024 -->  11 bits        11
0 to 250   =  251 -->   8 bits x 4 =  24
8 on/off                8 bits     =   8
                                    ----
                                      76 / 8 ~  10 Bytes
struct{
	uint16_t temp1;
	uint16_t temp2;
	uint16_t temp3_switchH;
	uint16_t temp4_switchL;
	uint8_t volt1;
	uint8_t volt2;
	uint8_t volt3;
	uint8_t volt4;
}payload;

void setup() {
  float temp1 =-25.0;
  float temp2 = 30.4;
  float temp3 = 125.1;
  int16_t temp4 = 999;

  float volt1 = 25.0;
  float volt2 = 4.5 ;
  float volt3 = 7.3;
  float volt4 = 0.5;
  uint8_t switch8 =0xAB;


  Serial.begin(115200);

  payload.temp1 = (uint16_t) ((temp1+25)*10);
  payload.temp2 = (uint16_t) ((temp2+25)*10);
  payload.temp3_switchH = (uint16_t) ((temp3+25)*10) + ((((uint16_t) switch8)<<8) & 0xF000); //(1251+25)d + 0xA000h(switch8 upper nibble)
  payload.temp4_switchL = (temp4+25) + (((uint16_t) switch8)<<12); //(999+25)d +0xB000h(switch8 lower nibble)
  payload.volt1 = (uint8_t) (volt1*10);
  payload.volt2 = (uint8_t) (volt2*10);
  payload.volt3 = (uint8_t) (volt3*10);
  payload.volt4 = (uint8_t) (volt4*10);

  Serial.println(sizeof(payload));
  Serial.println(payload.temp1);
  Serial.println(payload.temp2);
  Serial.println(payload.temp3_switchH);
  Serial.println(payload.temp4_switchL);
  Serial.println(payload.volt1);
  Serial.println(payload.volt2);
  Serial.println(payload.volt3);
  Serial.println(payload.volt4);

  
}

void loop() {
  // put your main code here, to run repeatedly:

}

output:

12 <--- payload byte size
0
554
42461
46080
250
45
73
5

hope that helps...

@MichelDeMeester You may want to have a look at bit fields.

struct Pack {
  uint16_t a : 11;  // 11 bit unsigned integer.
  uint16_t b : 5;   // 5 bit unsigned integer.
};

void setup() {
  Serial.begin(9600);

  Pack p {1000, 18};
  Serial.println(p.a);        // Prints "1000";
  Serial.println(p.b);        // Prints "18";
  Serial.println(sizeof(p));  // Prints "2" (bytes).
}

void loop() {}

[edit]

Combining post #14 and #17 into the following takes up 11 bytes (with room for four more bits).

struct Pack {
  uint16_t temp1 : 11;
  uint16_t temp2 : 11;
  uint16_t temp3 : 11;
  uint16_t temp4 : 11;
  uint16_t volt1 : 8;
  uint16_t volt2 : 8;
  uint16_t volt3 : 8;
  uint16_t volt4 : 8;
  uint16_t switch1 : 1;
  uint16_t switch2 : 1;
  uint16_t switch3 : 1;
  uint16_t switch4 : 1;
  uint16_t switch5 : 1;
  uint16_t switch6 : 1;
  uint16_t switch7 : 1;
  uint16_t switch8 : 1;
};
1 Like

Hello All,

I went for the combining 4 bits from the switches into the 4 msb's from the temp values.

Ending in exactly 12 bytes.

Thanks for thinking with me
Michel

1 Like

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.