How to Encode Binary Packets (COBS)?

Projects General Guidance
1

Hi all,

I want to COBS encode a binary packet containing 5 int32_t values to send over Bluetooth Serial on my ESP32. I am currently doing the following:

// Send binary to Bluetooth Serial Port
ESP_BT.write((byte*)&data1, sizeof(int32_t));
ESP_BT.write((byte*)&data2, sizeof(int32_t));
ESP_BT.write((byte*)&data3, sizeof(int32_t));
ESP_BT.write((byte*)&data4, sizeof(int32_t));
ESP_BT.write((byte*)&t_finish, sizeof(int32_t));
ESP_BT.write('\*');
ESP_BT.write('\*');
ESP_BT.write('\*');
ESP_BT.write('\*');

Where **** is the end-character however this is unreliable when decoding on the PC side. I am therefore trying COBS encoding/decoding using PacketSerial and their encode function:

    static size_t encode(const uint8_t* buffer,
                         size_t size,
                         uint8_t* encodedBuffer)
    {
        size_t read_index  = 0;
        size_t write_index = 1;
        size_t code_index  = 0;
        uint8_t code       = 1;

        while (read_index < size)
        {
            if (buffer[read_index] == 0)
            {
                encodedBuffer[code_index] = code;
                code = 1;
                code_index = write_index++;
                read_index++;
            }
            else
            {
                encodedBuffer[write_index++] = buffer[read_index++];
                code++;

                if (code == 0xFF)
                {
                    encodedBuffer[code_index] = code;
                    code = 1;
                    code_index = write_index++;
                }
            }
        }

        encodedBuffer[code_index] = code;

        return write_index;
    }

Issues:

  1. I don't know how to convert the in32_t data values into a uint8_t pointer array as the encode() input.
  2. How to pass the encodedBuffer to the ESP_BT.write() function.

Alternatively, should I be combining all 5 bytes into the buffer so that the COBS encoding creates a complete packet?

Apologies for my rudimentary questions, any hints are welcome!

2

This sounds strange. 5 longs means 20 byte.
Do You have problems with leading zero supression?
Are You sure You shouldn't send a delimiter after each long? Sending several delimiters after the 5 x longs looks strange to me.

3

Correct my packets should be 20 bytes long and I think I can get away with an end-delimiter since by data packets will always be a fixed size.

The standard \r\n delimiter shows up in my data, and with a 4 byte overhead (using ****) I have troubles with resynchronisation and bad packets on PC.

I want to try a binary encoding scheme and COBS seems like the most efficient and appropriate for my use case. I should mention I need 1000Hz through-put so byte overhead is a consideration here.

EDIT: jremington I mean 20 byte packets 1000 times a second (200,000 Baud) which I believe can be done via Bluetooth. Unfortunately, I cannot really sacrifice the Bluetooth or binary data communication, so I would really like to learn how this kind thing of communication is done reliably :smiley:

4

Is there a good reason to send binary data, or is this just intended as a learning experience?

Sending binary data usually creates more problems than it solves.

I need 1000Hz through-put

If you mean 1000 messages per second, don't use Bluetooth Serial. That is out of the question.

5

I should mention that I am referring to Bluetooth Classic on the esp32, which I understand correctly the default protocol reaches up to 2.1Mbps - although the transfer speed is higher than the actual throughput in practice.

I can also cut my data packets to 18 bytes by limiting the time stamp to int16_t for t_finish. COBS encoding supposedly adds a fixed overhead of 1 byte for packets less than 255, so even with a start and stop delimiter, shouldn't 20kBytes/s still be possible even over a BT classic communication protocol?

6

I suggest that you first figure out whether you can actually and reliably achieve the desired data rate, using 20 ASCII characters per packet. That is the equivalent of a serial Baud rate of about 200K. The result is readable and it is easy to see if there are transmission errors.

For example: "0123456789ABCDEFGHIJ"

7

Just did some testing to re-confirm why I moved to binary over serial a while back and I can say with confidence that there is simply too much delay in the transmission.

Part of my requirement is real-time plotting and sending ASCII characters is adding a lot of delay as I presume the BluetoothSerial transmit buffer is filling up quite fast. I should say that the int32_t data I am sending are raw 32-bit ADC values, so the ASCII overhead can increase rapidly when dealing with large numbers (which will occur).

I appreciate the advice but I am convinced sending binary packets is my only option, for which I'm hoping someone might be able to provide their experience with using COBS encoding.

8

If you can't send 20 ASCII characters at the required speed, then you can't send 20 binary bytes either. But it seems that this line of reasoning escapes you.

Good luck with your project!

9

This has nothing to do with being able to send 20 bytes of raw bytes/ ASCII code repetitively (which I can do) and everything to do with the fact that an int32_t in raw bytes will always be a fixed length of 4 bytes and ASCII code will not.

As I have already explained, I will have to transmit values between 0 – (2^31 - 1) and large numbers like these will cause significant overhead in ASCII code, compared to raw bytes.

Feel free to correct me if I am wrong in what I've said, but do not try to belittle me, since I am just trying to learn as much as any other person on this forum :slight_smile:

10

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.