Data packet bytes convert to float

[nikolaineda]

Hello,

I am trying to receieve a data packet of 70 bytes from one board to another with serial.

What i get when printing the characters will be shown in attached picture.
I believe these are ASCII symbols.
The baud rate has been checked and the data has been verified correct through RealTerm. So the board that sends data is really working properly.

My question is how can i transform these symbols into useable numbers since the packet is like this:
I would like to have a value of pressure that i can use for calculation further.

Byte index Description
0 Frame character '#' uint8
1
2 Pressure0 float32
3
4
...
9
10 Pressure2 float32
11
12
...

So each 4 bytes should represent one value of pressure.

Hope the code that i have been using is correct and that any of you could help me achieve final objective. I would like to have a value of pressure that i can use for calculation further.
Thanks in advance.

char buf[70];
String readString; 

#define read7hp Serial5
int readline(int readch, char *buffer, int len) {
    static int pos = 0;
    int rpos;

    if (readch > 0) {
        switch (readch) {
            case '\r': // Ignore CR
                break;
            case '\n': // Return on new-line
                rpos = pos;
                pos = 0;  // Reset position index ready for next time
                return rpos;
            default:
                if (pos < len-1) {
                    buffer[pos++] = readch;
                    buffer[pos] = 0;
                    readString += readch;
                }
        }
    }
    return 0;
}


union {
    float f;
    char c[4];
  } conv;

void setup() {
    Serial.begin(115200);
    read7hp.begin(115200);
}

void loop() {
    if (readline(read7hp.read(), buf, 70) > 0) {


        conv.c[0] = buf[33];
        conv.c[1] = buf[34];
        conv.c[2] = buf[35];
        conv.c[3] = buf[36];
        Serial.print("You entered: >");
        Serial.print(buf);
//        Serial.print(readString[33]);
//        Serial.print(readString[34]);
//        Serial.print(readString[35]);
//        Serial.print(readString[36]);

//        Serial.print(conv.f);
//        Serial.print(buf[1],HEX);
//        Serial.print(' ');
//        Serial.print(buf[2],HEX);
//        Serial.print(' ');
//        Serial.print(buf[3],HEX);
//        Serial.print(' ');
//        Serial.print(buf[4],HEX);
        Serial.println("<");
        readString="";
    }
    
    
}

[sterretje]

The received data is binary data; anything between from 0x00 to 0x1F and anything above 0x7F is basically not understood by the serial monitor and hence the 'funny' characters.

If you want to print the received data, loop through it and print using something like

for(byte cnt=0;cnt<70;cnt++)
{
  Serial.print(buffer[cnt], HEX);
  Serial.print(" ");
}
Serial.println();

Instead of printing the received data, print the float conv.f and check if that's what you expected.

Please share the spec of the other device or the code that generates the data.

[nikolaineda]

Hi,
Thanks for the reply.

I will attach one picture "data_packet" which describes the packet that is being send from another board. I cannot modify this since this board has been bought like that.

Also i will attach one more picture where you can see now the output with HEX format.

My main objective is that i want to have that float value for each pressure i receive (please look at data_packet graph), so that i can manipulate after with it.

It seems that union function from my code does not give reasonable values. In my union it transforms characters into float. In the end if i print conv.f i get random numbers and sometimes ovf.

Thanks for any further help

[horace]

do you know if the endianness of the source and destination processors are the same?
if not you will have to reversethe order of the 4 bytes which make up a 32bit float
e.g. a simple program looking at the frist pressure value after a #

union toFloat {
   unsigned char ch[4];
   float f;
   //} data= {0x28, 0x5F, 0x40, 0x40};    // gives 3.005
   } data= {0x40, 0x40, 0x5F, 0x28};  // gives 0.00

int main(void) {
  //data.f=3.14159;
  printf("%f", data.f);
  }

as giving the float is 3.005 when the order is reversed it gives a float value of 0.00
also could float be using different represations in the processors?

[nikolaineda]

Thanks.
Data are transmitted using the little endian convention.

After looking to output received on program RealTerm (direct Serial on Com for windows) i see that always each line begins with 23 which is HEX symbol for #. This is how it should be.

However in my case when i read, bytes and transfer them into HEX i have random symbol at the beginning.

Could you help me with an addition to the code where he starts reading when he sees first symbol #. I think after that it should be fine.

The current version of the code is:

char buf[70];


#define read7hp Serial5
int readline(int readch, char *buffer, int len) {
    static int pos = 0;
    int rpos;

    if (readch > 0) {
        switch (readch) {
            case '\r': // Ignore CR
                break;
            case '\n': // Return on new-line
                rpos = pos;
                pos = 0;  // Reset position index ready for next time
                return rpos;
            default:
                  if (pos < len-1) {
                    buffer[pos++] = readch;
                    buffer[pos] = 0;
                }
                  }        }
    return 0;
}

union {
    unsigned char c[4];
    float f;
  } conv;



void setup() {
    Serial.begin(115200);
    read7hp.begin(115200);
}

void loop() {
    if (readline(read7hp.read(), buf, 71) > 0) {



        conv.c[0] = buf[4];
        conv.c[1] = buf[3];
        conv.c[2] = buf[2];
        conv.c[3] = buf[1];
        
      for(byte cnt=0;cnt<70;cnt++)
      {
        Serial.print(buf[cnt], HEX);
        Serial.print(" ");
      }
//        Serial.println();
//        Serial.print("You entered: >");
//        Serial.print(buf);

//        Serial.printf("%f",conv.f);
//        Serial.print(buf[1],HEX);
//        Serial.print(' ');
//        Serial.print(buf[2],HEX);
//        Serial.print(' ');
//        Serial.print(buf[3],HEX);
//        Serial.print(' ');
//        Serial.print(buf[4],HEX);
        Serial.println("<");
    }
    
    
}

[horace]

can you post an example some Realterm output as text

[nikolaineda]

Ok first i will attach description of the packet that is being sent. Two graphs attached.

With the code from above i get something like this:

3E 68 DD A8 BD D2 9E C7 3D 3D 23 B0 6E 40 80 C8 3F F8 50 40 50 55 40 F8 42 40 A0 F9 3F 50 C6 3F A9 E8 C3 47 DF A6 B7 41 EB 51 EC 41 DE E3 41 D5 BC A2 BB 14 87 3F B7 11 97 3D 68 A1 BD A4 9D 5 3D 59 23 B0 6E 40 <
89 3C 70 9 19 BC DA 23 18 5F 40 A0 CD 3F E8 49 40 B0 48 40 E8 3C 40 F0 BF 3F E0 EF 3F BF E8 C3 47 F AC B7 41 EB 51 EC 41 38 E3 41 80 E6 BC 74 BB 68 87 3F DC 86 87 3C 68 9D 89 BD A4 3D 15 3D DA 23 88 5E 40 C0 <
89 3C 70 89 57 BC 94 23 70 6A 40 C0 DB 3F 8 4B 40 38 59 40 78 46 40 B0 D9 3F E8 3F A9 E8 C3 47 3C B1 B7 41 EB 51 EC 41 66 E3 41 80 F0 BC C4 BB 52 87 3F 24 79 EF BC 68 8D DF BD A4 5D 63 3D 18 23 20 70 40 70 D0 <
87 3F 92 1C 65 BD D0 BA 54 BD A4 BD 53 3D BC 23 18 5F 40 50 C2 3F E8 49 40 B0 5D 40 A8 36 40 40 D1 3F 10 D6 3F 3A E8 C3 47 D4 B3 B7 41 EB 51 EC 41 C0 E2 41 80 E4 BC A4 BB 6 87 3F C0 6D A8 3A A0 F5 ED BC 48 BB <
23 28 6A 40 80 CC 3F E8 52 40 68 72 40 F8 42 40 40 DA 3F 10 D6 3F 40 E8 C3 47 C8 C0 B7 41 EB 51 EC 41 66 E3 41 80 E7 BC A0 BB 36 87 3F 48 72 26 BC B4 4E 3 BE 80 4B E0 BA EF 23 F0 48 40 F0 BE 3F 70 55 40 E8 51 <
87 3F 90 E4 CF BB B4 A6 1E BE B8 C4 E8 BC 63 23 18 61 40 B0 DF 3F 30 4C 40 18 51 40 30 48 40 A0 D4 3F 40 F8 3F 2A E8 C3 47 61 C3 B7 41 EB 51 EC 41 D8 E2 41 C9 BC 90 BB 2A 87 3F B8 8D 50 3C D0 3A 16 BD E0 12 B5 <
89 3C A4 1D 44 3D 63 23 C8 64 40 40 ED 3F 20 4E 40 60 5F 40 48 40 40 F0 F2 3F 30 C5 3F 2A E8 C3 47 F9 C5 B7 41 EB 51 EC 41 D8 E2 41 80 E1 BC AE BB 36 87 3F 49 3E A9 BD A0 75 AF BC D2 9E C7 3D 5E 23 8 69 40 50 <
89 3C 70 9 19 BC 1F 23 8 69 40 50 DC 3F 60 5B 40 8 61 40 18 4F 40 50 F2 3F 30 DC 3F 2A E8 C3 47 61 C3 B7 41 EB 51 EC 41 8 E3 41 E1 BC 8E BB 22 87 3F DC C6 A6 3C A0 35 90 BC 48 FB EB 3C 23 28 6A 40 90 EB 3F <
3E 51 23 88 71 40 40 D7 3F 4F 40 20 64 40 78 4B 40 A0 CF 3F 80 CC 3F 2A E8 C3 47 56 D0 B7 41 EB 51 EC 41 A8 E2 41 E7 BC 7C BB 22 87 3F 49 6E A1 BD 68 8D DF BD D2 5E A8 3D 2C 23 40 6F 40 A0 E7 3F 61 40 E8 51 40 <

[Robin2]

nikolaineda:
I am trying to receieve a data packet of 70 bytes from one board to another with serial.

If you are sending the data from one Arduino to another Arduino why not just send it in human-readable form with Serial.print(). It will make the debugging much easier.

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data. There is also a parse example to illustrate how to extract numbers from the received text.

The technique in the 3rd example will be the most reliable. It is what I use for Arduino-to-Arduino communication.

You can send data in a compatible format with code like this

Serial.print('<'); // start marker
Serial.print(value1);
Serial.print(','); // comma separator
Serial.print(value2);
Serial.println('>'); // end marker

I would only consider the inconvenience of sending data in binary form if it was essential for performance purposes.

...R

[nikolaineda]

Robin2:
If you are sending the data from one Arduino to another Arduino why not just send it in human-readable form with Serial.print(). It will make the debugging much easier.

Thanks for your reply. Unfortunately the board that is sending data has been bought as is. So its not me who designed the packet. My only objective is to make sure to collect it on another board so that i can use it with other instruments and do some more calculations.

[horace]

you appear to assume your data is fterminated by '\n'

          case '\n': // Return on new-line
                rpos = pos;
                pos = 0;  // Reset position index ready for next time
                return rpos;

you then attempt to process it
yet the frame description specifies it starts with #, followed by 17 float values (4 bytes each) and terminates with a 1 byte checksum
can you post a sample of the raw data fom the device (connect it directly to Realterm)?

[cattledog]

Do you have control over the data format? If so, you can follow the tutorial methods of Robin2.

What I am not clear about from the formamt documents and data you posted (since there are 4byte float values included) if 0x23 and 0x0A are not part of the data stream as well.

In the data you posted, there are multiple 0x23 values not 70 bytes apart.

[horace]

if I read it correctly after each # (0x23) there should be 68 bytes of data (17*4) and 1 checksum
I have been attempting to count bytes in attached file HEX_output.png in post #2

what happens at startup if the data contains a 0x23 ?
I guess one looks for a start of frame 0x23, reads the frame and calculates the checksum
if OK process the received data else start again

[Robin2]

nikolaineda:
Thanks for your reply. Unfortunately the board that is sending data has been bought as is.

Do you have the specification for what the board sends? If so post a link to it.

Without the specification you should order a lot of coffee

...R

[nikolaineda]

Robin2:
Do you have the specification for what the board sends? If so post a link to it.

Without the specification you should order a lot of coffee

...R

Please see the attachments in post #6. This is the description of byte packet that i would like to transfrom into useable values that i can after manipualte with.

horace:
if I read it correctly after each # (0x23) there should be 68 bytes of data (17*4) and 1 checksum
I have been attempting to count bytes in attached file HEX_output.png in post #2

what happens at startup if the data contains a 0x23 ?
I guess one looks for a start of frame 0x23, reads the frame and calculates the checksum
if OK process the received data else start again

That is completely correct.
The raw data in HEX from realTerm looks like:

2300E843400080CC3F006044400040474000181F400000BC3F0020C33FD887C247DA59AF411E85E741003EE1410000C5BC0060A73D00D6873FB88D503C68EDCFBDD21E893DD7

23005854400080A43F00C8224000002D400008254000509E3F0050E03F5D88C2476F5CAF411E85E7410014E2410080B9BC00E0A63D00E0873F24B991BC68BDD7BDB8C4E8BC1E

2300C055400070B43F0090314000C0494000783D400080DD3F0060D73F1B88C2479661AF411E85E741005AE2410000C0BC0080A53D0086873FB80D123C685DE7BDA43D153DE0

cattledog:
Do you have control over the data format? If so, you can follow the tutorial methods of Robin2.

What I am not clear about from the formamt documents and data you posted (since there are 4byte float values included) if 0x23 and 0x0A are not part of the data stream as well.

In the data you posted, there are multiple 0x23 values not 70 bytes apart.

Please see attachments in post #6. There is no control in the format of data i want to receive.

[Robin2]

Images from Reply #6 so we don't have to download them. See this Simple Image Guide

...R

[Robin2]

So it seems that you need a function that ignores everything until a '#' is received and then saves the next 69 bytes.

I don't know if the float32 is what an Arduino uses for a float (I rarely use floats). If it is then you can create a struct with 17 floats and one byte (for the checksum) and read the data directly into it and there will be no need to parse anything.

...R

[GolamMostafa]

nikolaineda:
The raw data in HEX from realTerm looks like:
2300E843400080CC3F006044400040474000181F400000BC3F0020C33FD887C247DA59AF411E85E741003EE1410000C5BC0060A73D00D6873FB88D503C68EDCFBDD21E893DD7

23005854400080A43F00C8224000002D400008254000509E3F0050E03F5D88C2476F5CAF411E85E7410014E2410080B9BC00E0A63D00E0873F24B991BC68BDD7BDB8C4E8BC1E

2300C055400070B43F0090314000C0494000783D400080DD3F0060D73F1B88C2479661AF411E85E741005AE2410000C0BC0080A53D0086873FB80D123C685DE7BDA43D153DE0

1. I have manually added all the bytes (except the last one) of the 1st frame; the normal checksum (the upper part = the accumulated carries have been discarded) has appeared as D7. I have done the same process for the 2nd frame, and the checksum has appeared as 1E. Now, it is confirmed that they are valid telemetry frames with 0x23 as the Frame Marker (beginning of a new Frame) followed by 68-byte 'binary formatted data' and then followed by 1-byte checksum.

2. The algorithm to catch the frame could be like this:

A: Read the byte (the frameMarker) from serial port and store it into x
if(x == 0x23)
{
   Read next 69-byte
   Calculate checksum for the first 69-byte including the 1st byte (frameMarker = 0x23)
   if (checksum == 70th byte)
   {
      This is the correct frame   //in-spite of multiple 0x23s within 70-byte data
      Compute Pressure and display.
   }
   else
   {
      Begin from Label A
   }
}
else
{
   Begin from Label A
}

3. Arduino Coding of the algorithm of Step-2
Sender UNO Codes

#include <SoftwareSerial.h>
SoftwareSerial mySUART (4, 5); //SRX, STX
byte sendData[] = 
{
  0x23, 0x00, 0xE8, 0x43, 0x40, 0x00, 0x80, 0xCC,
  0x3F, 0x00, 0x60, 0x44, 0x40, 0x00, 0x40, 0x47, 
  0x40, 0x00, 0x18, 0x1F, 0x40, 0x00, 0x00, 0xBC, 
  0x3F, 0x00, 0x20, 0xC3, 0x3F, 0xD8, 0x87, 0xC2, 
  0x47, 0xDA, 0x59, 0xAF, 0x41, 0x1E, 0x85, 0xE7, 
  0x41, 0x00, 0x3E, 0xE1, 0x41, 0x00, 0x00, 0xC5, 
  0xBC, 0x00, 0x60, 0xA7, 0x3D, 0x00, 0xD6, 0x87, 
  0x3F, 0xB8, 0x8D, 0x50, 0x3C, 0x68, 0xED, 0xCF, 
  0xBD, 0xD2, 0x1E, 0x89, 0x3D, 0xD7
};

void setup()
{
  Serial.begin(9600);
  mySUART.begin(9600);
}

void loop()
{
  for (int i = 0; i < sizeof(sendData); i++)
  {
    Serial.print(sendData[i], HEX);
    mySUART.write(sendData[i]);
  }
  Serial.println();
  Serial.println("================================");
  delay(2000);
}

Figure-1: Serial Monitor of sender UNO

Receiver NANO Codes

#include<SoftwareSerial.h>
SoftwareSerial mySUART(4, 5); //SRX, STX
bool flag1 = LOW;
byte recData[68] = "";
byte frameMarker;
byte chkSumRec = 0;
byte chkSumCal=0;
int i = 0;

void setup()
{
  Serial.begin(9600);
  mySUART.begin(9600);
}

void loop()
{
  if (mySUART.available() > 0)
  {
    if (flag1 == HIGH)
    {
      recData[i] = mySUART.read();
      Serial.print(recData[i], HEX);
    //  chkSum += recData[i];
      i++;
      if (i == 68)
      {
        chkSumRec = mySUART.read();
        //chkSum = chkSum - mySUART.read();
        //if (chkSum == 0x00)
        //{
      //  while(1);
        Serial.println();
        Serial.print("Received chkSumRec: ");
        Serial.println(chkSumRec, HEX); //received chkSumRec
        chkSumCal = frameMarker;
        for(int i = 0; i<sizeof(recData); i++)
        {
          chkSumCal += recData[i];
        }
        Serial.println();
        Serial.println("Calculated chkSumCal = ");//Valid frame is received!");
        Serial.println(chkSumCal, HEX);
        

        i = 0;
        flag1 = LOW;
        Serial.println();
        Serial.println("=================================");
      } else {}
    }
    else
    {
      byte x = mySUART.read();
      if (x == 0x23)
      {
        Serial.println("Marker is found!");
        frameMarker = x;
        flag1 = HIGH;
      }
    }
  }
}

Figure-2: Serial Monitor of receiver NANO

4. Now, the frame is well received and all the 4x17 (68-byte) data for 16 different pressures are available in recData[] buffer of the Receiver NANO. You can now process this data using union structure to find the individual pressures an display them.
Hints:

union
{
   byte press_0Array[4];
   float press_0;
}pressure_0;

for(int i =0; i<3; i++)
{
  pressure_0.press_0Array[i] = recData[i];     //assume receied bytes are little endian oriented
}

Serial.println(pressure_0.press_0, 2); //Pressure_0 = xx.xx units (mbar?)

[nikolaineda]

GolamMostafa:
....

4. Now, the frame is well received and all the 4x17 (68-byte) data for 16 different pressures are available in recData[] buffer of the Receiver NANO. You can now process this data using union structure to find the individual pressures an display them.
Hints:

union

{
  byte press_0Array[4];
  float press_0;
}pressure_0;

for(int i =0; i<3; i++)
{
  pressure_0.press_0Array[i] = recData[i];    //assume receied bytes are little endian oriented
}

Serial.println(pressure_0.press_0, 2); //Pressure_0 = xx.xx units (mbar?)

Wouuu! Awesome job. It works nicely!! Thank you very much!!!