guide/library for messaging protocol for radio

sbaratheon · January 9, 2021, 4:24pm

Hi

I'm currently working with some LoRa radios which are being used to transmit and receive various information. At the moment, I'm developing my own messaging codes to accompany certain information (e.g. sending a tx with "REQUEST_SETTINGS" which the other radio may respond to with "MODE = 20, SPEED = 3.321").

I'm still working on this but I have to admit that it is rather tiresome having to parse each message and to develop all of the messaging codes. I was therefore wondering whether there are any libraries which do most of the heavy lifting for me? To be clear, I have a LoRa library which interfaces with the LoRa device but that is at a fairly low level (e.g. send message with string) and I was wondering whether there is something at a higher level.

If not, please could someone recommend some reading as to what considerations I should take into account when developing the messaging (I'm fairly new to networking/protocols so any guidance would be great).

Thanks

mikb55 · January 9, 2021, 5:20pm

You can avoid parsing by serialising/deserialising a C struct. (subject to LoRa packet size limits)
You'll still need to write code to do the 'if request is this, then send this'.

Robin2 · January 9, 2021, 5:22pm

sbaratheon:
(e.g. sending a tx with "REQUEST_SETTINGS"

Why not just send 'S' for settings? Single characters make the parsing simple.

...R

sbaratheon · January 9, 2021, 8:30pm

mikb55:
You can avoid parsing by serialising/deserialising a C struct. (subject to LoRa packet size limits)
You'll still need to write code to do the 'if request is this, then send this'.

thanks, completely forgot about some form of binary serialisation!

srnet · January 9, 2021, 10:06pm

Be careful using structures to send and receive data across different Arduino platforms, the structure construction or padding can be different.

Its not clear to me what your question is about, are you just after an easier way of filling a LoRa packet with a mixture of variable types (including arrays) or looking for some standardised way of passing sensor type data such as in the cayenne format ?

sbaratheon · January 10, 2021, 12:23am

srnet:
Be careful using structures to send and receive data across different Arduino platforms, the structure construction or padding can be different.

Its not clear to me what your question is about, are you just after an easier way of filling a LoRa packet with a mixture of variable types (including arrays) or looking for some standardised way of passing sensor type data such as in the cayenne format ?

Hi
Yes, I've just been looking at binary serialisation and it seems that padding/endianness can cause issues.
I'm basically looking for a simple way to transfer plain old data (PODs) structures via LoRa. For example, I want to transfer this struct (among others):

struct settings
{
    uint32_t waitPeriodSecs = 5;
    uint32_t epoch = 838382;
    char callsign[10] = "station283";
};

Is there a relatively simple way of doing this? At the moment, I've just using a simple data format which is a pain to work with e.g.

WAIT_PERIOD_SECS=5, EPOCH=838382, CALLSIGN=station283

Thanks

jremington · January 10, 2021, 2:11am

Is there a relatively simple way of doing this?

Nothing could be simpler. Just send the struct as a packet, like you would any other packet.

The receiver has to know all about the struct, of course, taking into account the limitations noted in reply #4.

The following will fail because it takes 11 bytes to store the C-string "station283", including the required zero terminator.

char callsign[10] = "station283";

srnet · January 10, 2021, 8:41am

sbaratheon:
Is there a relatively simple way of doing this? At the moment, I've just using a simple data format which is a pain to work with e.g.
WAIT_PERIOD_SECS=5, EPOCH=838382, CALLSIGN=station283

Cant see how, apart from sending a packet for each variable.

A LoRa packet is just a series of bytes and the receiver has no way of knowing how to decode these bytes into meaningful data unless you define the variables you want to extract. A structure is the clearest and most obvious (when reading a program listing) way of how to achieve this.

Whilst I have produced example programs for my LoRa library using structures to send and receive data, I avoid structures in general as you cannot be sure the example will work across platforms.

On method I adopted to avoid the use of structures was to build a packet like this (it uses specific Library functions);

  LoRa.startWriteSXBuffer(0);                         //start the write at SX12XX internal buffer location 0
  LoRa.writeBufferChar(trackerID, sizeof(trackerID));     //+4 bytes (3 characters plus null (0) at end)
  LoRa.writeFloat(latitude);                          //+4 = 8 bytes
  LoRa.writeFloat(longitude);                         //+4 = 12 bytes
  LoRa.writeUint16(altitude);                         //+2 = 14 bytes
  LoRa.writeUint8(satellites);                        //+1 = 15 bytes
  LoRa.writeUint16(voltage);                          //+2 = 17 bytes
  LoRa.writeInt8(temperature);                        //+1 = 18 bytes total to send
  TXPacketL = LoRa.endWriteSXBuffer();                //closes packet write and returns the length of the packet to send

At the receiver you need to read out the variables in the same type and order;

  LoRa.startReadSXBuffer(0);               //start buffer read at location 0
  LoRa.readBufferChar(receivebuffer);      //read in the character buffer
  latitude = LoRa.readFloat();             //read in the latitude
  longitude = LoRa.readFloat();            //read in the longitude
  altitude = LoRa.readUint16();            //read in the altitude
  satellites = LoRa.readUint8();           //read in the number of satellites
  voltage = LoRa.readUint16();             //read in the voltage
  temperature = LoRa.readInt8();           //read in the temperature
  RXPacketL = LoRa.endReadSXBuffer();      //finish packet read, get received packet length

Accepted its not as clear as using a structure, but it has the very significant advantage of being platform agnostic.

One additional benefit is that the routines above do not use a memory buffer\array for building the packet, variables are written direct to the LoRa devices internal buffer.

The previously mentioned Cayenne uses identifier bytes to describe the type of data that follows, voltage, temperature, GPS co-ordinate etc. Thus the receiver does not actually need to know the structure or variable type of the data sent.

sbaratheon · January 10, 2021, 10:47am

jremington:
Nothing could be simpler. Just send the struct as a packet, like you would any other packet.

I'm not sure that I understand what you mean. The LoRa packet must be a byte/char array and so the question arises as to how a struct is serialised to said byte/char array.
As noted above, it seems that a simple binary serialisation using memcpy is unreliable as between different arduino architectures.

sbaratheon · January 10, 2021, 10:58am

srnet:
Cant see how, apart from sending a packet for each variable.

A LoRa packet is just a series of bytes and the receiver has no way of knowing how to decode these bytes into meaningful data unless you define the variables you want to extract. A structure is the clearest and most obvious (when reading a program listing) way of how to achieve this.

Whilst I have produced example programs for my LoRa library using structures to send and receive data, I avoid structures in general as you cannot be sure the example will work across platforms.

On method I adopted to avoid the use of structures was to build a packet like this (it uses specific Library functions);
  LoRa.startWriteSXBuffer(0);                         //start the write at SX12XX internal buffer location 0

LoRa.writeBufferChar(trackerID, sizeof(trackerID)); //+4 bytes (3 characters plus null (0) at end)
LoRa.writeFloat(latitude); //+4 = 8 bytes
LoRa.writeFloat(longitude); //+4 = 12 bytes
LoRa.writeUint16(altitude); //+2 = 14 bytes
LoRa.writeUint8(satellites); //+1 = 15 bytes
LoRa.writeUint16(voltage); //+2 = 17 bytes
LoRa.writeInt8(temperature); //+1 = 18 bytes total to send
TXPacketL = LoRa.endWriteSXBuffer(); //closes packet write and returns the length of the packet to send

At the receiver you need to read out the variables in the same type and order;
LoRa.startReadSXBuffer(0); //start buffer read at location 0
LoRa.readBufferChar(receivebuffer); //read in the character buffer
latitude = LoRa.readFloat(); //read in the latitude
longitude = LoRa.readFloat(); //read in the longitude
altitude = LoRa.readUint16(); //read in the altitude
satellites = LoRa.readUint8(); //read in the number of satellites
voltage = LoRa.readUint16(); //read in the voltage
temperature = LoRa.readInt8(); //read in the temperature
RXPacketL = LoRa.endReadSXBuffer(); //finish packet read, get received packet length
Accepted its not as clear as using a structure, but it has the very significant advantage of being platform agnostic. 

One additional benefit is that the routines above do not use a memory buffer\array for building the packet, variables are written direct to the LoRa devices internal buffer.

The previously mentioned Cayenne uses identifier bytes to describe the type of data that follows, voltage, temperature, GPS co-ordinate etc. Thus the receiver does not actually need to know the structure or variable type of the data sent.

Thanks, that's helpful. One other method might be using a text serialisation library e.g. [ArduinoJson](http://"Thanks, that's helpful. One other method might be using a text serialisation library e.g. ArduinoJson.").

jremington · January 10, 2021, 9:07pm

The LoRa packet must be a byte/char array and so the question arises as to how a struct is serialised to said byte/char array.

There is no such requirement. Use a cast to get around assumptions in your code.

Post the code you use to send a packet, and we can show you how to modify the call to send anything.

srnet · January 10, 2021, 9:18pm

sbaratheon:
The LoRa packet must be a byte/char array and so the question arises as to how a struct is serialised to said byte/char array.

I agree with @jremington, since you can write direct to the buffer the LoRa device internally uses to send a packet, there is no requirement for the data you are writing (and sending as a packet) to be in an 'array' at all.

nicolajna · January 10, 2021, 9:59pm

You can do something like this

void serializePacket(struct settings *s, uint8_t *data) {

	data[0] = (uint8_t)(s->waitPeriodSecs >> 24);
	data[1] = (uint8_t)(s->waitPeriodSecs >> 16);
	data[2] = (uint8_t)(s->waitPeriodSecs >> 8);
	data[3] = (uint8_t)s->waitPeriodSecs;

	data[4] = (uint8_t)(s->epoch >> 24);
	data[5] = (uint8_t)(s->epoch >> 16);
	data[6] = (uint8_t)(s->epoch >> 8);
	data[7] = (uint8_t)s->epoch;

	uint8_t n = 8;
	uint8_t i;
	for (i = 0; i < 10; i++) {
		data[n++] = s->callsign[i];
	}
}

void deserializePacket(struct settings *s, uint8_t *data) {

	s->waitPeriodSecs = (uint32_t)((data[0] << 24) | (data[1] << 16) | (data[2] << 8) | data[3]);
	s->epoch = (uint32_t)((data[4] << 24) | (data[5] << 16) | (data[6] << 8) | data[7]);

	uint8_t n = 8;
	uint8_t i;
	for (i = 0; i < 10; i++) {
		s->callsign[i] = data[n++];
	}
}

I've not compiled this or tested this at all, but it should give you an idea of how you could do it. This takes care of the endianness issue. It also result in a much smaller payload than your text based approach.

sbaratheon · January 10, 2021, 10:16pm

jremington:
There is no such requirement. Use a cast to get around assumptions in your code.

Post the code you use to send a packet, and we can show you how to modify the call to send anything.

Ok, thanks. I use the following to send a message:

bool sendText(const char* msg)
{
    DEBUG("Sending Message: ");
    DEBUG_NL(msg);
    
    if (strlen(msg) > MAX_LORA_PACKET)
    {
      return false;
    }
  
    if ( !remoteStationRadio.beginPacket() )
    {
        return false;
    }

    remoteStationRadio.print(msg); // prints up to, but not including, the null symbol

    if ( !remoteStationRadio.endPacket() ) 
    {
        return false;
    }
    return true;
}

I'm using this library in case it helps.

jremington · January 10, 2021, 10:27pm

One of many approaches to sending a struct would be to replace this line in the above

    remoteStationRadio.print(msg); // prints up to, but not including, the null symbol

with

    remoteStationRadio.write( (uint8_t *) &name_of_struct, sizeof(name_of_struct)); // send the struct

(this actually works for any type of variable, not just structs).

Substitute "name_of_struct" with whatever you call your struct. Of course, you will need to make the struct accessible to the sending function, either in the argument list, or as a global variable.

I would probably write a separate function, named something like sendBinary(), along the lines of the existing function sendText(), but with an additional argument giving the message size in bytes.

sbaratheon · January 10, 2021, 11:08pm

jremington:
One of many approaches to sending a struct would be to replace this line in the above
    remoteStationRadio.print(msg); // prints up to, but not including, the null symbol
with
    remoteStationRadio.write( (uint8_t *) &name_of_struct, sizeof(name_of_struct)); // send the struct
(this actually works for any type of variable, not just structs).

Substitute "name_of_struct" with whatever you call your struct. Of course, you will need to make the struct accessible to the sending function, either in the argument list, or as a global variable.

I would probably write a separate function, named something like sendStruct(), along the lines of the existing function sendText(), but with an additional argument giving the message size in bytes.

Thanks very much.
As I understand it, this effectively casts the struct (or any other object) to a byte array. For my own learning, please could you explain how this avoids the padding issue (srnet's post #4 above)?

jremington · January 10, 2021, 11:15pm

Padding is only an issue if you are mixing processor architectures, which leads to many other problems as well. An "int" could have one, two or four bytes, for example.

If you plan on using several different node architectures, then a human readable text-based message protocol can be cost effective, and it makes debugging and long term data storage easier.

You might even consider XML, for which there is an Arduino parser called tinyXML.

system · May 10, 2021, 11:16pm

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Heltec WiFi LoRa 32 (V3) - Encode/decode packets to binary (make message shorter) Networking, Protocols, and Devices	36	2704	September 22, 2024
Understanding LoRa packet Networking, Protocols, and Devices	40	5511	August 6, 2024
Heltec LoRa 32 v3.2: Receive a Structure Variable w/ heltec_unofficial library Programming	63	832	October 18, 2025
Send and Receive numerical data (int) via LoRa General Guidance	4	1620	February 18, 2024
Snprintf clarification General Guidance	22	240	October 17, 2025

guide/library for messaging protocol for radio

Related topics