How do you combine sensor data with custom protocol und send it with Serial?

Hey guys,

I am very new to Arduino.

Now I am trying to send the sensor data to Computer.

The sensor data will be stored as bytes into arrays. To build a robust transmission I'll pack the data with own defined protocol. e.g. has Header, length, Checksum ...

My question is, how to combine the sensor data with protocol frames and send it with serial.

Now I have written something like below, it works, but very simple and dirty :slight_smile: . So I'd like to know how you handle my case?

#define SOF_1 0x55
#define SOF_2 0xAA

uint8_t SensorDataByteArrA[4] = {0xc1, 0xc2, 0xc3, 0xC4};
uint8_t SensorDataByteArrB[4] = {0xe1, 0xe2, 0xe3, 0xe4};

void send_sensor_data()
{
  Serial.write(SOF_1);
  Serial.write(SOF_2);

  Serial.write(SensorDataByteArrA, 4);
  Serial.write(SensorDataByteArrB, 4);

}

have a look at Arduino serial basics
what is receiving the serial information? e.g. a human reader or a program
a humer reader will require readable text, a program can receive raw binary (your example looks like binary data transmission)

to send binary you could use a frame along the lines of (with DLEs being byte stuffed)
SYN DLE STX byte stuffed data CRC DLE ETX

horace:
have a look at Arduino serial basics
what is receiving the serial information? e.g. a human reader or a program
a humer reader will require readable text, a program can receive raw binary (your example looks like binary data transmission)

to send binary you could use a frame along the lines of (with DLEs being byte stuffed)
SYN DLE STX byte stuffed data CRC DLE ETX

Hi, other program will receive the message and parse it into readable text. Using Frame is exactly what I want. But I would to learn some "elegant" ways to put all the bytes fragments into a complete Frame. Using Serial.write() in a send-function again and again is really too dirty

if you are using a serial communications channel you end up with Serial or something similar
why not use WiFi or Ethernet to send UDP datagrams or set up a TCP virtual circuit?

Then there is sprintf() (or snprintf()). Arduino disables floats for sprintf() but you can turn floats to strings with dtostrf() to use with sprintf().

But there is nothing really wrong with many prints or writes (with delimiters between).

If the messages are always the same length you can statically prepare (initialize) the message invariants in a character array with place markers for the variable data. Then just update the correct locations in the array as you need, then send it all out with one write statement.

sun0727:
Hi, other program will receive the message and parse it into readable text. Using Frame is exactly what I want. But I would to learn some ‚Äúelegant‚ÄĚ ways to put all the bytes fragments into a complete Frame. Using Serial.write() in a send-function again and again is really too dirty

Why would that be elegant? Just so you perform the send once? If you have variable length frames, you would have to allocate memory sufficient for the largest expected frame. That could be wasteful. What about the frame assembly itself? Now you have to shuffle data around and all that. Not necessarily in your case, but it is a common beginners mistake to not even consider the possibility of sending things separately. Sending the data shouldn‚Äôt be that ‚Äúdirty‚ÄĚ, just an indexing loop. First you find the length, calculate the CRC. Now you send framing, length, data, CRC and framing. The only case where it makes a lot of sense, is where the medium itself demands packetization, such as ASK RF transmission.

I don’t like the bare print class in Arduino core, I use a library to enable streaming, so I can do things like:

  console << VT_GREEN
          << F("LED Control Utility") << ' ';
  console  << endl << F("available commands:") << VT_DEFAULT_COLOUR << endl;

If you can use binary, you can create a structure (theStruct)…
fill the data values,
‚ÄĒ theStruct.valueX = valueX;
calculate the checksum
‚ÄĒ theStruct.chksum = (calculatedValue);
then simply
‚ÄĒ Serial.write(theStruct)

You know what’s coming, so on the receiving side, take the binary serial bytes into a char array, map that to a structure like that in the sender, and there’s your data packet.

Time to tell us whether your data has a fixed or variable length. As you should be able to see, it makes a difference. You posted a sample, but who knows how preliminary it is?

For my ASK RF comms, I used a struct as mentioned above.

(quite old, YMMV)...

// WeatherPacket.h

struct weatherPacket_t
{
  // 4 bytes
  bool isWeatherPacket : 1; //Packet is a weather message
  uint8_t station_conditions : 7; // 83 condition codes
  bool isGusting : 1; //Wind gusts exist
  uint8_t station_humidity : 7; // 0-100
  
  bool isWarning : 1; //Warnings exist

  // time zone * 4
  int8_t timezone : 7; // range can be from +14h (Line Islands Time) to -11h (Samoa Standard Time)

  uint8_t station_windspeed;  // 0-255 kph

  // 4*2 = 8 bytes
  unsigned int station_bearing : 9; // 1-360 degrees
  unsigned int station_visibility : 7; // km * 4
  int station_temperature;    // degrees C * 256
  unsigned int station_pressure;  // kPa * 256
  int local_temperature;    // degrees C * 256

  // 4 bytes
  unsigned long time;
};

Notice that I was able to fit multiple fields into standard integer types, using the "bit field" feature. This was more efficient than wasting, for example, an entire byte just to transmit 2 bits.

aarg:
Time to tell us whether your data has a fixed or variable length. As you should be able to see, it makes a difference. You posted a sample, but who knows how preliminary it is?

For my ASK RF comms, I used a struct as mentioned above.

(quite old, YMMV)...

// WeatherPacket.h

struct weatherPacket_t
{
  // 4 bytes
  bool isWeatherPacket : 1; //Packet is a weather message
  uint8_t station_conditions : 7; // 83 condition codes
  bool isGusting : 1; //Wind gusts exist
  uint8_t station_humidity : 7; // 0-100
 
  bool isWarning : 1; //Warnings exist

// time zone * 4
  int8_t timezone : 7; // range can be from +14h (Line Islands Time) to -11h (Samoa Standard Time)

uint8_t station_windspeed;  // 0-255 kph

// 4*2 = 8 bytes
  unsigned int station_bearing : 9; // 1-360 degrees
  unsigned int station_visibility : 7; // km * 4
  int station_temperature;    // degrees C * 256
  unsigned int station_pressure;  // kPa * 256
  int local_temperature;    // degrees C * 256

// 4 bytes
  unsigned long time;
};




Notice that I was able to fit multiple fields into standard integer types, using the "bit field" feature. This was more efficient than wasting, for example, an entire byte just to transmit 2 bits.

Using struct to pack a frame is really a good tips for me. Could you please share more about how do you put the data into the "struct" and send them? Thank you so much

struct member access is done using the dot operator, for example I would say:

weatherPacket_t weather;
...
weather.isWeatherPacket = true;
weather.station_humidity = DHT.humidity(); // fake sensor call just for example

RadioHead transmits arrays of uint8_t. So you have to cast or "pun" the struct to that data type in order to send it. There are different approaches to that, IIRC there are some examples given in the RH library, I could be wrong. Here is a post from not too long ago, on a slightly different subject, which might still have some suggestion:

Reply #7 is basically it for structs, make an attempt, show your code, and we can work from there...
If you try, you’ll get more out of it than us doing the work.

lastchancename:
Reply #7 is basically it for structs, make an attempt, show your code, and we can work from there...
If you try, you’ll get more out of it than us doing the work.

Thank you so much. Below is my code. From my point of view, it "almost" works. Only some bytes are reversed. e.g. I have set the SOF to 0x55AA, but I got 0xAA55 in serial monitor. The data bytes are also reversed. But I think it is due to Serial.write.

As a newbie, I would like to know whether this is a way to transfer data. I have used ''malloc'' to store sensor data with dynamic length. But some articles says, using "malloc" in MCU can cause unexpected problem... Anyway, any advice would be appreciated :slight_smile:

#define SERIAL_BAUDRATE 9600  // serial baudrate


//=== custom message protocol
#define SOF   0x55AA         // Start of Frame
#define SOF_1 0x55
#define SOF_2 0xAA
#define SENSOR_ALL  0xEE     // use to mark when all sensors are active  

//=== hypothetical sensor results
#define CHECK_SUM   0xBB    // hypothetical checksum

float hypoTemp = -12.34;    // hypothetical temperature from sensor 0xC14570A4
float hypoHumi = 45.67;     // hypothetical Humidity from sensor  0x4236AE14 


// ==== Define Data packet
typedef struct {
  uint16_t  sof;        //  start of frame
  uint8_t   state;      //  sensor state
  uint8_t   len;        //  length of sensor data
  uint8_t   data[];     //  flexible array to store data
}arduinoDataPacket;


//==== union type for float sensor data
typedef union {
 float floatSensorData;
 uint8_t byteArrSensorData[4];
} unionSensorData;


// global variable to store temperature data und humidity data
unionSensorData unionTemp;
unionSensorData unionHumi;

// combine 4 Bytes Temperature Data + 4 Byates humidity Data
uint8_t sensorDataArr[8];



void setup() 
{
    Serial.begin(SERIAL_BAUDRATE);

    delay(1000);
    Serial.println("Finish Setup");
    //send_sensor_data();
}


void loop() 
{
    // put your main code here, to run repeatedly:
    Serial.println("hello");          // HEX 0x68 65 6c 6c 6f
    delay(1000);

    send_data_packet();  
    delay(500);
}



void send_data_packet()
{
    // declare data packet 
    arduinoDataPacket *packetToSend;

    update_sensor_data();                       // update sensor data to gloabl variable
    
    uint8_t dataLen = sizeof(sensorDataArr);    // length of data frame
    uint8_t sensorState = SENSOR_ALL;           // byte to mark the State of Sensor

    create_sensor_data_packet(SENSOR_ALL, sensorDataArr, dataLen, &packetToSend);

    uint8_t packetLen = sizeof(arduinoDataPacket) +  dataLen + 1  // 1 byte for Checksum
    Serial.write((uint8_t *) packetToSend, packetLen);  // byte-wise send

    // get {AA 55 EE 08 A4 70 45 C1 14 AE 36 42 BB}

    free(packetToSend);  // free memory

}


void create_sensor_data_packet(uint8_t state, uint8_t *sensorData, 
                               uint8_t dataLen, arduinoDataPacket **endPacket)
{

    arduinoDataPacket *frame = NULL;
    uint8_t frameLen = 0;

    // check parameter
    if (NULL == endPacket || ((NULL == sensorData) && (dataLen != 0)))
    {
        return 0;
        Serial.println("Parameter Error!");
    }

    
    frameLen = sizeof(arduinoDataPacket) + dataLen + 1;  // 1 for checksum

    // allocate the required memory to store data packet
    frame = (arduinoDataPacket *)malloc(frameLen);

    memset((uint8_t *)frame, 0x00, frameLen);   // init. the memory with 0
    
    // write frames to packet
    frame->sof = SOF;
    frame->state = state;
    frame->len = dataLen;

    if(sensorData && dataLen)
    {
        memcpy(frame->data, sensorData, dataLen);
    }

    frame->data[dataLen] = CHECK_SUM;    //the last byte in data frame is Checksum

    //Serial.write((uint8_t *)frame, frameLen);

    *endPacket = frame;     // endPacket point to frame

}


void update_sensor_data()
{
    update_bme280_temperature();
    update_bme280_humidity();

    //uint8_t sensorDataArr[8]; // fix size

    uint8_t *tempDataArr = unionTemp.byteArrSensorData;
    uint8_t *humiDataArr = unionHumi.byteArrSensorData;

    // copy to global variable sensorDataArr
    memcpy(sensorDataArr, tempDataArr, 4);  
    memcpy(sensorDataArr + 4, humiDataArr, 4);
}


void update_bme280_temperature()
{
    // --- get temperature data from senor----
    // ... reserve ...

    // --- save to unionTemp ----
    unionTemp.floatSensorData = hypoTemp;   // using hypothetical temperature at this time
}


void update_bme280_humidity()
{
    // --- get temperature data from senor----
    // ... reserve ...

    // --- save to unionTemp ----
    unionHumi.floatSensorData = hypoHumi;   // using hypothetical humidity at this time

}

aarg:
struct member access is done using the dot operator, for example I would say:

weatherPacket_t weather;

...
weather.isWeatherPacket = true;
weather.station_humidity = DHT.humidity(); // fake sensor call just for example




RadioHead transmits arrays of uint8_t. So you have to cast or "pun" the struct to that data type in order to send it. There are different approaches to that, IIRC there are some examples given in the RH library, I could be wrong. Here is a post from not too long ago, on a slightly different subject, which might still have some suggestion:

https://forum.arduino.cc/index.php?topic=704058.msg4732233#msg4732233

Thank you so much. I have post my current Code in #12. Any advice would be appreciated :slight_smile: !

I haven’t looked too closely (it’s 4am), but I’m not sure you need all the fuss around your formatting and malloc() etc.
If you need different packet types, simply have as many structure types, and send them with a that identifies what type/ size packet is coming next.

I’ll look closer if I find time later today.

lastchancename:
I haven’t looked too closely (it’s 4am), but I’m not sure you need all the fuss around your formatting and malloc() etc.
If you need different packet types, simply have as many structure types, and send them with a that identifies what type/ size packet is coming next.

I’ll look closer if I find time later today.

oh my god, you are sooo warm-hearted! You don’t owe me anything… Just have a nice rest!