How to interpret 4 bytes in reverse as float without memcpy?

I am reverse engineering a proprietary packet based
protocol and working on a device to intercept it. For that I am receiving the packets one by one, storing them temporarily in a variable and then forwarding them.
But between storing and forwarding I also want to manipulate these packets and I can't really find an elegant way of doing this.

The most obvious way of course would be to store everything as a uint8_t array and then read/manipulate the individual bytes. But some of the data in some packets is stored in little endian and needs to be interpreted as float.

For the moment I have created this crazy overly complicates struct:

        struct User {
            uint8_t startSequence[2];   // 2 bytes for START SEQUENCE
            uint8_t bodySize;           // 1 byte for BODY SIZE
            uint8_t type;               // 1 byte for TYPE
            uint8_t unknown;            // 1 byte for UNKNOWN
            uint8_t entityId;           // 1 byte for ENTITY ID
            uint8_t reserved[3];        // 3 bytes for RESERVED
            std::vector<uint8_t> value; // Variable size for VALUE
            uint8_t endSequence[2];     // 2 bytes for END SEQUENCE

            User(const uint8_t* data, size_t length) {
                if (length < 11) {
                    throw std::invalid_argument("Data too short to be a valid User packet");
                }

                // Parse header
                startSequence[0] = data[0];
                startSequence[1] = data[1];
                bodySize = data[2];
                type = data[3];
                unknown = data[4];

                if (type != Types::USER) {
                    throw std::invalid_argument("Invalid type for User packet");
                }

                // Parse body
                entityId = data[5];
                reserved[0] = data[6]; 
                reserved[1] = data[7];
                reserved[2] = data[8];
                value.assign(data + 9, data + length - 2);

                // Parse footer
                endSequence[0] = data[length - 2];
                endSequence[1] = data[length - 1];

                if (startSequence[0] != START_SEQUENCE[0] || startSequence[1] != START_SEQUENCE[1]) {
                    throw std::invalid_argument("Invalid start sequence");
                }

                if (endSequence[0] != END_SEQUENCE[0] || endSequence[1] != END_SEQUENCE[1]) {
                    throw std::invalid_argument("Invalid end sequence");
                }
            }

            /**
             * This function returns the value of the packet as a boolean.
             */
            bool getBoolValue() const {
                if (value.size() != 1) {
                    throw std::invalid_argument("Value size is not 1");
                }

                return value[0] == 0x01;
            }

            /**
             * This function returns the value of the packet as an unsigned 8-bit integer.
             */
            uint8_t getUint8Value() const {
                if (value.size() != 1) {
                    throw std::invalid_argument("Value size is not 1");
                }

                return value[0];
            }

            /**
             * This function returns the value of the packet as a 32-bit float in little-endian format.
             */
            float getFloatLeValue() const {
                if (value.size() != 4) {
                    throw std::invalid_argument("Value size is not 4");
                }

                // Reorder the bytes to form the correct little-endian format
                uint32_t temp = (value[0]) |
                                (value[1] << 8) |
                                (value[2] << 16) |
                                (value[3] << 24);
                float result;
                memcpy(&result, &temp, sizeof(result));
                return result;
            }
        };

And even though this works, I am rsther dissatisfied with it because I'm sure there is a way cleaner way of doing this.

Ideally I would be able to be able to just map my data packet into the struct and be able to do sth like

User* packet = reinterpret_cast<User*>(data);
packet->value = 3.518;

In a way that the original byte array is correctly updated with the little endian representation for these 4 bytes.

All the Arduino CPUs are natively little-endian, so you should be able to just memcpy() from your packet into your float.
Theoretically, the compiler is smart enough to optimize such memcpy use...

3 Likes

This invokes Undefined Behavior and breaks the optimizer's TBAA.

See Don't use unions or pointer casts for type punning for alternatives.

2 Likes

Please, explain your objectives with the help of a binary32 formatted 4-byte data.

Is there a type-accessibility violation here if both the raw data and the struct are a bunch of uint8_t (separately because they're both a bunch of the same stuff, and that stuff is a typedef for unsigned char)?

However, for that same reason, you can't avoid the memcpy

  • it's needed as "barrier" to do the aforementioned type punning without Undefined Behavior
  • the value offset in the struct is not 32-bit-aligned, so it may not be valid to access it directly as a float
  • on the plus side, the mechanics of the copy will be as optimized as possible for the platform

I got a direct cast to work (apparently)

#include <stdexcept>

// struct UserBase {
//   virtual bool boolValue() const = 0;
//   virtual uint8_t uint8Value() const = 0;
//   virtual float floatValue() const = 0;
// };

template<typename T>
struct User { //: UserBase {
  uint8_t startSequence[2];   // 2 bytes for START SEQUENCE
  uint8_t bodySize;           // 1 byte for BODY SIZE
  uint8_t type;               // 1 byte for TYPE
  uint8_t unknown;            // 1 byte for UNKNOWN
  uint8_t entityId;           // 1 byte for ENTITY ID
  uint8_t reserved[3];        // 3 bytes for RESERVED
  uint8_t value[sizeof(T)];   // Variable size for VALUE -- `T value` would not be 32-bit-aligned
  uint8_t endSequence[2];     // 2 bytes for END SEQUENCE

  bool boolValue() const {
    return uint8Value();  // or does it have to be 0x01 exactly?
  }
  uint8_t uint8Value() const {
    if (sizeof(value) != 1) {
      throw std::invalid_argument("Value size is not 1");
    }

    return value[0];  
  }
  float floatValue() const {
    if (sizeof(value) != 4) {
      throw std::invalid_argument("Value size is not 4");
    }

    float result;
    memcpy(&result, value, sizeof(result));
    return result;
  }
};

void setup() {
  Serial.begin(115200);
  uint8_t data1[] = {
    0, 1,
    2, 3, 4, 5,
    6, 7, 8,
    9,
    10, 11,
  };
  auto *packet1 = reinterpret_cast<User<uint8_t>*>(data1);
  uint8_t data4[] = {
    0, 1,
    2, 3, 4, 5,
    6, 7, 8,
    0x44, 0x33, 0x22, 0xc1,
    13, 14,
  };
  auto *packet4 = reinterpret_cast<User<float>*>(data4);
  Serial.println(packet1->boolValue());
  Serial.println(packet1->uint8Value());
  Serial.println(packet4->floatValue());
}

void loop() {
  delay(10);
}

Because the data is variable-size in the middle, you need distinct types, especially if you want to cast it back to raw data. The decision for the exact type, previously based on length, is done manually in the demo.

Note there is a commented-out UserBase, in an attempt to handle the results polymorphically. The reinterpret_cast does not fail (it can't), but the function calls don't work, probably because with virtual methods, the class instances have a reference to their vtable, and it's not in the raw data.

If you want to handle these generically, you'll need to add something, depending on their usage.

1 Like

@kenb4 I see, thanks for the explanation.
@westfw Does this apply to the ESP32 as well?

Would it help if I went for a struct with fixed value size instead?

I tried it like this:

#include <Arduino.h>

typedef struct {
    uint8_t start_seq[2];
    uint8_t body_size;
    uint8_t type;
    uint8_t unknown;
    uint8_t entity_id;
    uint8_t reserved[3];
    float value;
    uint8_t end_seq[2];
} Packet;

void printRawPacket(const char* message, const uint8_t* packet, size_t length) {
    Serial.print(message);
    for (int i = 0; i < length; i++) {
        Serial.printf("%02X ", packet[i]);
    }
    Serial.println();
}

// Packet interceptor function
void handlePacket(uint8_t* raw_packet, size_t length) {
    // Create a pointer to the packet structure
    Packet* packet = (Packet*)raw_packet;

    printRawPacket("Original bytes: ", raw_packet, sizeof(raw_packet));
    // Correctly Prints: "Original bytes: BE EF 08 03 00 09 00 00 00 00 00 C0 40 EF BE"

    // Access the float value without creating a copy of the 4 bytes
    Serial.print("Original value: ");
    Serial.println(packet->value);
    // Incorrectly prints a random value or "ovf" instead of 6.00

    // Modify the float value without creating a copy of the 4 bytes
    packet->value = -120.0f; // little endian would be 00 00 F0 C2; big endian would be C2 F0 00 00
    Serial.print("Modified value: ");
    Serial.println(packet->value); // Prints -120.0 correctly

    printRawPacket("Modified bytes: ", raw_packet, sizeof(raw_packet));
    // Incorrectly Prints: "Modified bytes: BE EF 08 03 00 09 00 00 00 00 00 C0 00 00 F0"
    // Showing the float value was not stored correctly and the footer was overwritten
    // Expected output would have been: "Modified bytes: BE EF 08 03 00 09 00 00 00 00 00 00 00 F0 C2 EF BE"
}

void setup() {
    Serial.begin(115200);

    // Example packet
    uint8_t raw_packet[] = {
        // ------ HEADER ------
        0xBE, 0xEF, // start sequence
        0x08,       // body size uint8_t
        0x03,       // type uint8_t
        0x00,       // unknown
        // ------ HEADER ------

        // ------ BODY ------
        0x09,                   // entity id uint8_t
        0x00, 0x00, 0x00,       // reserved
        0x00, 0x00, 0xC0, 0x40, // value (float, little-endian) // 6.0f
        // ------ BODY ------

        // ------ FOOTER ------
        0xEF, 0xBE // end sequence
        // ------ FOOTER ------
    };

   handlePacket(raw_packet, sizeof(raw_packet));
}

void loop() {
    // Nothing to do here
}

But as mentioned in the comments, it's not working right.

The only way I was able to get it to work correctly, was by introducing a getter and setter again:

typedef struct {
    uint8_t start_seq[2];
    uint8_t body_size;
    uint8_t type;
    uint8_t unknown;
    uint8_t entity_id;
    uint8_t reserved[3];
    uint8_t value[4];
    uint8_t end_seq[2];

    // Function to get the float value from the raw packet in little-endian format
    float getFloatLeValue() const {
        // reverse byte order and return as float
        uint32_t temp = (value[3] << 24) | (value[2] << 16) | (value[1] << 8) | value[0];
        float result;
        memcpy(&result, &temp, sizeof(result)); // Copy uint32_t to float
        return result;
    }

    // Function to set the float value into the raw packet in little-endian format
    void setFloatLeValue(float newValue) {
        // Copy float to uint32_t
        uint32_t temp;
        memcpy(&temp, &newValue, sizeof(temp));
        // Copy uint32_t to value in little-endian format
        value[0] = temp & 0xFF;
        value[1] = (temp >> 8) & 0xFF;
        value[2] = (temp >> 16) & 0xFF;
        value[3] = (temp >> 24) & 0xFF;
    }
} Packet;

But this just seems unnecessarily complex and complicated when all I wanna do is interpret 4 bytes in reverse.

As mentioned by @PieterP, that cast is illegal and may cause undefined behavior.

Also, on an ESP32 your Packet type and raw_packet array are different sizes due to alignment by the compiler.

BTW, typedef is sooo 1980's 'C' language. Don't need it to define a struct type in C++.

Try this:

#include <Arduino.h>

struct __attribute__((packed)) Packet {
  uint8_t start_seq[2];
  uint8_t body_size;
  uint8_t type;
  uint8_t unknown;
  uint8_t entity_id;
  uint8_t reserved[3];
  float value;
  uint8_t end_seq[2];
};

void printRawPacket(const char* message, Packet* packet) {
  Serial.print(message);
  uint8_t *rawBytes = reinterpret_cast<uint8_t *>(packet);
  for (int i = 0; i < sizeof(Packet); i++) {
    Serial.printf("%02X ", rawBytes[i]);
  }
  Serial.println();
}

// Packet interceptor function
void handlePacket(Packet* packet) {

  printRawPacket("Original bytes: ", packet);

  Serial.print("Original value: ");
  Serial.println(packet->value);

  // Modify the float value without creating a copy of the 4 bytes
  packet->value = -120.0f;
  Serial.print("Modified value: ");
  Serial.println(packet->value);

  printRawPacket("Modified bytes: ", packet);
}

void setup() {
  Serial.begin(115200);
  delay(2000);

  Packet tempPacket;
  // Example packet
  uint8_t raw_packet[] = {
    // ------ HEADER ------
    0xBE, 0xEF, // start sequence
    0x08,       // body size uint8_t
    0x03,       // type uint8_t
    0x00,       // unknown
    // ------ HEADER ------

    // ------ BODY ------
    0x09,                   // entity id uint8_t
    0x00, 0x00, 0x00,       // reserved
    0x00, 0x00, 0xC0, 0x40, // value (float, little-endian) // 6.0f
    // ------ BODY ------

    // ------ FOOTER ------
    0xEF, 0xBE // end sequence
    // ------ FOOTER ------
  };

  Serial.println(sizeof(raw_packet));
  Serial.println(sizeof(Packet));
  memcpy(&tempPacket, raw_packet, sizeof(Packet));
  handlePacket(&tempPacket);
}

void loop() {
}

Result:

15
15
Original bytes: BE EF 08 03 00 09 00 00 00 00 00 C0 40 EF BE 
Original value: 6.00
Modified value: -120.00
Modified bytes: BE EF 08 03 00 09 00 00 00 00 00 F0 C2 EF BE 
1 Like

You cannot use sizeof to get the number of elements in an array if all you have is a pointer to it.

This is equivalent to the reinterpret_cast you had before, it also invokes Undefined Behavior. See also: Don't use C-style casts

1 Like

@gfvalvo Thank you, to my surprise, this actually works. So I guess float on ESP32 has always been little-endian after all? Also thanks for showing me the modern way of doing it.

@PieterP Thanks for clearing that up. I was genuinely not aware of that.

BTW, this:

is one of the few cases where you're allowed to type pun via a pointer. You can cast to a uint8_t * in order to access the individual bytes in the original data structure.

1 Like

Usually, float stands for "Floating Point Number" like 0.15625 (for example). In ESP32, a floating point number can be stored as --

1. 4-byte (single precision) binary32/IEE-754 formatted data (Fig-2.1, little endian) by declaring:

float myNum = 0.15625;

IEEE754Float
Figure-2.1:

2. 8-byte (double precision) binary64/EE-754 formatted data (Fig-2.2, little endian) by declaring:

double myNum = 1.23456;

binary64
Figure-2.2:

1 Like

Do you have any suggestions on how to design several different packet types without too much redundancy while keeping the order intact?

For example I would like to do something among the lines of:

struct __attribute__((packed)) GenericPacket {
    uint8_t start_seq[2]; // always {0xBE, 0xEF}
    uint8_t body_size;
    uint8_t packet_type;
    uint8_t header_byte;
    uint8_t body[body_size];
    uint8_t end_seq[2]; // always {0xEF, 0xBE}
};

struct __attribute__((packed)) GenericEntityPacket : public GenericPacket {
    // Replace uint8_t body[body_size]; with:
        uint8_t entity_id;
        uint8_t reserved[3];
        uint8_t remaining_body[body_size-4];
};

// Example: BE EF 08 03 00 58 00 00 00 00 00 C0 40 EF BE
struct __attribute__((packed)) FloatEntityPacket : public GenericEntityPacket {
    // Replace `uint8_t remaining_body[body_size-4];` with
        float value;
};

// Example: BE EF 05 03 00 59 00 00 01 EF BE
struct __attribute__((packed)) BoolEntityPacket : public GenericEntityPacket {
    // Replace `uint8_t value[body_size-4];` with
        uint8 unused[3];
        bool value;
};

// Example: BE EF 05 03 00 5C 00 00 00 04 EF BE
struct __attribute__((packed)) Uint8tEntityPacket : public GenericEntityPacket {
    // Replace `uint8_t value[body_size-4];` with
        uint8 unused[3];
        uint8_t value;
};

// Example: BE EF 01 18 01 02 EF BE
struct __attribute__((packed)) ToggleTunerPacket : public GenericPacket {
    // Replace `uint8_t header_byte;` with:    
        uint8_t state;
    // Replace `uint8_t body[body_size];` with
        uint8_t channel;
};

// Example: BE EF 00 1E 02 EF BE
struct __attribute__((packed)) SdRecordPacket : public GenericPacket {
    // Replace `uint8_t header_byte;` with:    
        uint8_t state;
    // Remove `uint8_t body[body_size];`
};

// Example: BE EF 01 1A 01 04 EF BE
struct __attribute__((packed)) SnapshotPacket : public GenericPacket {
    // Replace `uint8_t header_byte;` with:    
        uint8_t action;
    // Replace `uint8_t body[body_size];` with
        uint8_t slot;
};

Edit: Moved to How to elegantly inherit from struct overriding/adding fields

Hi @felic

One way to represent 4-bytes as a floating point number without memcpy is to use a union:

// Test: set float to the smallest number greater than 1
union
{
  uint8_t byteData[4];        // byte array
  float f;                    // float
  struct
  {
    uint32_t mantissa : 23;   // 23 bit mantissa
    uint32_t exponent : 8;    // 8 bit exponent
    uint32_t sign     : 1;    // 1 bit sign
  } part;
} singleFloat;

void setup()
{
  SerialUSB.begin(115200);                // Start communication on the native USB port
  while(!SerialUSB);                      // Wait for the console to open 
  singleFloat.part.mantissa = 0x1;        // Load mantissa
  singleFloat.part.exponent = 0x7F;       // Load exponent 
  singleFloat.part.sign = 0x0;            // Load sign 
  for (uint8_t i = 0; i < 4; i++)
  {
    SerialUSB.print(F("Byte"));
    SerialUSB.print(i);
    SerialUSB.print(F(": "));
    SerialUSB.println(singleFloat.byteData[i], HEX);
  }
  SerialUSB.print(F("Float: "));
  SerialUSB.println(singleFloat.f, 8);   // Should output: 1.00000012
}

void loop() {}

This is not allowed either. See the link posted earlier: Don't use unions or pointer casts for type punning

2 Likes

Hi @PieterP

Thanks for the link to the information about type punning.

Looks like my suggestion fell foul of the C++ type system.

Yes. There hasn't been a big-endian CPU released on a long time.
(Some of the larger ARM, MIPS, and PPC processors have an OPTION (usually selectable at boot time) to run either little-endian or big-endian, but all of the microcontrollers I've seen have been little-endian.
(I guess there are some Motorola/Freescale/NXP "CPU32" (~68k-like) chips that might qualify as microcontrollers, and are still big-endian. They're not seen very often, though.)

The banning of assorted types of type-punning without an "adequate" replacement is something I find annoying. Those were widely-used techniques in networking and embedded code that AFAIK worked fine for decades.
"Just use memcpy() and the compiler will optimize it, maybe" is a poor substitute.

2 Likes

I agree. It makes sense from a compiler developer's perspective, but it has some serious downsides for networking and deserialization code.

C++ now finally has std::start_lifetime_as, but there's not a single compiler that supports it yet. Meanwhile, most other C++23 features are supported by at least some of the major compilers, so either this is a low-priority feature, or it's hard to implement. Probably both. Compiler support for C++23 - cppreference.com

The endian-ness of floating point numbers is the same as (multi-byte) integers for a given CPU. If it helps to remember: the sign bit is in the same place, at the high bit.

Intel's C compiler has a bi-endian data feature.
Declaring a variable as "big endian" means that the compiler will insert a BSWAP instruction after each memory fetch. Since BSWAP is essentially a single-cycle instruction, it's much quicker than any memory fetch (even one from cache), and has negligible impact on performance.

It's rather neat. In theory, it could be added to other compilers. ARM also has a BSWAP-like instruction. (more performance impact on slower CPUs with more rightly coupled RAM, of course.)

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.