How to best read in Serial packet, parse out several data, rinse, repeat?

Newly minted program.

And this time I read ALL the notes on the program and got past my compile issue caused by println (changed println(recievedBytes) to println((char*)recievedBytes).

const byte numBytes = 32;
byte receivedBytes[numBytes];

boolean newData = false;


void setup() {                              // Put your setup code here, to run once

  Serial.begin(19200);                      // Start serial connection
  while (!Serial);                          // Not really sure what this is for, but examples have it...

}


void loop() {                               // put your main code here, to run repeatedly
  
  recvWithStartEndBytes();                  // ...
  showNewData();                            // ...

}


void recvWithStartEndBytes() {
  
  static boolean recvInProgress = false;
  static byte ndx = 0;                      //
  byte startByte = 10000000;                // <- start byte
  byte endByte = 01000000;                  // <- stop byte
  byte rb;                                  //
  
  while (Serial.available() > 0 && newData == false) {
   rb = Serial.read();

   if (recvInProgress == true) {
     if (rb != endByte) {
       receivedBytes[ndx] = rb;
       ndx++;
       if (ndx >= numBytes) {
         ndx = numBytes - 1;
       }
     }
     else {
       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;
       ndx = 0;
       newData = true;
     }
    }

    else if (rb == startByte) {
      recvInProgress = true;
    }
  }
  
}


void showNewData() {
  if (newData == true) {
    Serial.print ("This just in ...");
    Serial.println((char*)receivedBytes);
    newData = false;
  }
}

Ran it off of the serial transmission from my sensors and on the serial monitor all I got was line after line of:

This just in...

When I unplugged the serial Rx, the arduino would stop sending "This just in...", but with it plugged in, I didn't seem to get the additional "recievedBytes" packet printed as I expected to.

These are wrong
byte startByte = 10000000; // <- start byte
byte endByte = 01000000; // <- stop byte
They should be
byte startByte = 0b10000000; // <- start byte
byte endByte = 0b01000000; // <- stop byte
to tell the compiler it is binary data

Alternatively just use
byte startByte = 128; // <- start byte
byte endByte = 64; // <- stop byte

And modify this piece of code to show how many bytes are received each time

       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;
       Serial.print("Num Bytes ");    // <---- new
       Serial.println(ndx);                 // <---- new
       ndx = 0;
       newData = true;

...R

I was just about to try:

byte startByte = B10000000;

after reading this:

https://www.arduino.cc/en/Reference/Byte

but I went ahead and used "128" and "64" instead.

As far as the new code/program output, now I get this over and over:

This just in ...
Num Bytes 5

The fact that it's sending and re-sending both these lines, as well as showing a numBytes not equal to 32 means that something's being captured, just not printed back over serial.

While it would be nice to see what the arduino sees for a packet, I think I'll try to parse the individual bytes from the packet and see what I can do/make with that and print back over serial.

You need to give me an example of a complete packet as it is transmitted. Then I can do some experiments myself.

If you are expecting to receive 32 bytes you should increase the receive array to (say) 35 bytes to make sure there is no chance of overflow or missing something.

...R

Haven't had time to work on it today (I goofed on a circuit design and am scrambling to make the best of a very expensive mistake), BUT I do have a TXT file from Putty of some output from a few months back. It's in ASCII, but with some work can be translated to whatever.

Let me know if you'd like it attached until I can get my Arduino to output something.

And thank you for all the help.

Or if there is some way to accept in serial over the Rx pin and output it directly to the serial monitor, I can provide output from there as well.

Flinkly:
BUT I do have a TXT file from Putty of some output from a few months back. It's in ASCII, but with some work can be translated to whatever.

Three points ...
If the file contains the stuff that the Arduino program is expected to receive it is what I want. But it would be important to identify what is considered to be a single message so I can program an Uno to send it.

When you say it is in ASCII it make me wonder if that is the format that the Arduino will be receiving? I need to be able to emulate the exact data that your Arduino will be receiving. If you just mean that it is the ASCII values of characters (65 instead of 'A') that is fine.

If there are non-printing characters (such as CR and LF) in the data I need to be able to see them.

Attaching a text file will be fine - hope it is not too long.

...R

So I'm a little proud of myself, because I got output finally.

I tried a few things like "Serial.write()", but I finally just added a "Serial.println(receivedBytes[ndx], BIN)" as the program placed each byte received over serial into the receivedBytes[] array.

From this I received:

0
0
0
1
110111

The first three bytes are expected to be zero for this sensor setup, and the last two bytes are the data. I'm slightly bothered that the output doesn't show all 8 "digits" of the full bytes, but I'm sure that's something I can tell the Arduino to do if I knew how.

Again, here is the protocol of the input I'm EXPECTING to be receiving:

Sensor Address MSB
Sensor Address LSB // <- the sensor connected has an address value of zero
Sensor Instance // <- only one of this type of sensor is connected, so this is again zero
Sensor Data MSB // Now I just need to figure out what bitwise conversion is needed for real data
Sensor Data LSB //

I think I'll connect my second sensor now so that I can make my program able to differentiate between different sensors and apply conversions to the data as necessary. I think I am starting to get the hang of this program now.

Code attached for completeness of post:

//#include <Wire.h>

//#include "Adafruit_LEDBackpack.h"
//#include "Adafruit_GFX.h"

const byte numBytes = 32;
byte receivedBytes[numBytes];

boolean newData = false;

//Adafruit_7segment matrix = Adafruit_7segment();


void setup() {                              // Put your setup code here, to run once

  Serial.begin(19200);                      // Initialize serial connection
  while (!Serial);                          // Not really sure what this is for, but examples have it...

  //matrix.begin(0x70);                       // Initialize 7Segment
  //matrix.print(10000, DEC);                 // Print "----" to 7Segment
  //matrix.writeDisplay();                    // Write the previous Print to the 7Segment

}


void loop() {                               // put your main code here, to run repeatedly
  
  recvWithStartEndBytes();                  // ...
  //showNewData();                            // ...

}


void recvWithStartEndBytes() {
  
  static boolean recvInProgress = false;
  static byte ndx = 0;                      //
  byte startByte = 128;                     // <- start byte
  byte endByte = 64;                        // <- stop byte
  byte rb;                                  //
  
  while (Serial.available() > 0 && newData == false) {
   rb = Serial.read();

   if (recvInProgress == true) {
     if (rb != endByte) {
       receivedBytes[ndx] = rb;
       
       Serial.println(receivedBytes[ndx], BIN);
       
       ndx++;
       if (ndx >= numBytes) {
         ndx = numBytes - 1;
       }
     }
     else {
       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;

       Serial.println();                         //adds a carriage return after each complete packet

       //Serial.print("Num Bytes ");
       //Serial.println(ndx);

       //Serial.print("This just in ...");
       //Serial.write(receivedBytes, );
       
       //matrix.println(ndx);
       //matrix.writeDisplay();
       
       ndx = 0;
       newData = true;
     }
    }

    else if (rb == startByte) {
      recvInProgress = true;
    }
  }
  
}


void showNewData() {
  if (newData == true) {
    //Serial.print("This just in ...");
    //Serial.write(receivedBytes, );
    newData = false;
  }
}

Flinkly:
Again, here is the protocol of the input I'm EXPECTING to be receiving:

As I said earlier if you provide a sample of the data I will try some code myself.

...R

Ok, found a few terminal programs and one worked pretty well and I got two outputs (one hex, the other ascii) and a screen capture of binary data.

Also, I connected the second sensor, so this is twice as large of a data packet as I have been receiving so far. So this data would create a 10 item array instead of the 5 item array I have been getting when testing the program earlier.

capture_HEX.txt (9.8 KB)

capture_ascii.txt (21.8 KB)

I can't open your file capture_ascii.txt and I can't make sense of capture_HEX.txt.

Can you just extract a single message from the file and post that.

...R

Yeah, so for the HEX output, "80" is the start byte (Decimal - 128, Binary - 10000000) and "40" is the stop byte (Decimal - 64, Binary - 01000000). So a packet in HEX is:

[ 800000000220000300051500040000003F040000003F0000000040 ]

EDIT - Actually, you're right, the HEX output doesn't make any sense. Too many bytes of data for what a packet should be.

EDIT 2 - ACTUALLY, there is a few things going on here that I wasn't expecting, but I believe it's all correct (and kind of makes sense).

First, I only have two sensors, but I have two displays from the same company as well, and it appears that they are submitting their own packet of information to tell other displays down the line that they have already displayed the data for one of the sensors (so you could have two of the same sensor, and two displays showing that KIND of data, and they wouldn't mix the data up). So, a packet for a single sensor/display is 5 bytes, and we're receiving 5 sets of 5 bytes (2 sensors, 2 displays, and see next paragraph...).

Second, the extra data packet is because my second sensor captures vacuum AND pressure data, so it is (apparently) special and sends two sets of data, one for the vacuum side of the scale, the other for the pressure side of the scale (one or the other sends zero data, depending on what the sensor is reading). so that is where the "extra" set of data is coming from.

So while a lot of this isn't described in the protocol document I have (which isn't really that helpful), I think I get it.

The sensor with the MSB/LSB address of:

00 00
-is an oxygen sensor

00 03
-is a vacuum sensor

00 04
-is a pressure sensor (it's the same physical sensor as the vacuum sensor)

3F 04
-is the display showing the vacuum/pressure sensor data (this section of data might change between 3F 03 and 3F 04 depending on whether the sensor is reading Vacuum or Pressure at that moment)

3F 00
-is the display showing the oxygen sensor data

So overall, this program is going to be a little more work than I imagined. Also, I'm making this program to replace the displays that I currently have, so while this data is accepting a packet with 5 sets of data in it, I'll only be using it to collect 3 sets of data. I can collect data without the displays in-line, and would have if I had known they were adding their own information.

Either way, can still collect this data in the array and just parse out the desired information regardless of what other "data" is coming in the packet.

Sorry, but your description of different displays and whatnot is just confusing me.

Let's try to do one part at a time.

Before I do any programming can you confirm that this is a typical message that you need to receive.
800000000220000300051500040000003F040000003F0000000040
and there are no characters before or after that stream of hex data - such as a CR or LF

And there is something that you have not yet clarified - is that a series of characters 8 0 0 0 etc or is it the HEX representation of the bytes 128 0 0 0 2 32 etc

It would also be useful if you post a version of that message with spaces in it to signify the different groups of bytes that need to be extracted.

When we can receive and parse that data we can consider other issues.

...R

This is a HEX representation of the Bytes of a typical packet, and there aren't any LF or CR. Here is the data packet with spaces inserted between the HEX numbers:

80 00 00 00 02 20 00 03 00 05 15 00 04 00 00 00 3F 04 00 00 00 3F 00 00 00 00 40

While a whole packet of typical data is 27 bytes (25 information bytes and a start and stop byte), WITHIN the packet are groupings of 5 bytes (shown below):

80
00 00 00 02 20
00 03 00 05 15
00 04 00 00 00
3F 04 00 00 00
3F 00 00 00 00
40

Within each grouping of 5 bytes of data, 4 bytes are important to capture for my use, the first two and the last two bytes.

The first two bytes of a grouping of 5 bytes tell me what kind of data is coming.

The last two bytes of a grouping of 5 bytes are the actual raw data.

Does that explain the packet well enough?

Flinkly:
Does that explain the packet well enough?

Thanks. That seems OK.
I will have a look at it in the morning.
...R

I have a present for you :wink:

This should handle the framing and decoding.

#define TESTING_   // comment for real test

#ifdef TESTING_
const byte testData[] = {
  0x1F, 0x00, 0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00,
  0x00, 0x3F, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x00,
  0x00, 0x40, 0x80, 0x00, 0x00, 0x00, 0x02, 0x20, 0x00, 0x03,
  0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x04,
  0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x40, 0x80,
  0x00, 0x00, 0x00, 0x02, 0x20, 0x00, 0x03, 0x00, 0x05, 0x15,
  0x00, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x04, 0x00, 0x00, 0x00,
  0x3F, 0x00, 0x00, 0x00, 0x00, 0x40, 0x80, 0x00, 0x00, 0x00,
  0x02, 0x20, 0x00, 0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00,
  0x00, 0x00, 0x3F, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x00, 0x00,
  0x00, 0x00, 0x40, 0x80, 0x00, 0x00, 0x00, 0x02, 0x1F, 0x00,
  0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x3F,
  0x04, 0x00, 0x00, 0x00
};

byte testByte;
#endif

const byte startByte = 0x80;
const byte endByte = 0x40;

bool newPacket;

byte packet[25];
byte contains;
bool started;

#ifdef TESTING_
int test_available() {
  return sizeof(testData) - testByte;
}

byte test_read() {
  if (testByte < sizeof(testData)) {
    return testData[testByte++];
  } else {
    return 0;
  }
}
#endif

void processPacket();
void processSerial();

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (newPacket) {
    processPacket();
    newPacket = false;
  }
#ifdef TESTING_
  if (test_available()) {
#else
  if (Serial.available()) {
#endif
    processSerial();
  }
}

int twelveBits(byte* pMsbLsb) {
  return ((pMsbLsb[0] & 0x3F) << 6) | (pMsbLsb[1] & 0x3F);
}

void processPacket() {
  Serial.println("Packet");
  for (byte sensor = 0; sensor < 25; sensor += 5) {
    int adr = twelveBits(&packet[sensor]);
    int data = twelveBits(&packet[sensor + 3]);
    Serial.print("Adr 0x"); Serial.print(adr, 16);
    Serial.print(" Inst "); Serial.print(packet[sensor + 2]);
    Serial.print(" Data 0x"); Serial.println(data, 16);
  }
}

void processSerial() {
  do {
#ifdef TESTING_
    byte newByte = test_read();
#else
    byte newByte = Serial.read();
#endif
    if (!started) {
      started = (newByte == startByte);
    } else {
      switch (newByte) {
        case startByte:
          break;
        case endByte:
          newPacket = (contains == 25);
          started = false;
          break;
        default:
          if (contains < 25) {
            packet[contains++] = newByte;
            continue;
          }
          started = false;
      }
      contains = 0;
      break;
    }
#ifdef TESTING_
  } while (test_available());
#else
  } while (Serial.available());
#endif
}

Result (from embedded testdata):

Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0
Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0
Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0

I have written a short program for an Uno to transmit the data

And I have made a few changes to the example "recvWithStartEndMarkers" from Serial Input Basics so that it works with bytes rather than chars. I also added an extra function so that it will display the data in groups of 5 as well as on one line.

For simplicity I have used a Mega to receive the data as I can use Serial1 and still display the data on the Serial Monitor.

The connections between the two Arduinos are GND to GND and Uno Tx to Mega Rx1.

Uno transmit program

byte dataToSend[] = {0x80 ,0x00 ,0x00 ,0x00 ,0x02 ,0x20 ,0x00 ,0x03 ,0x00 ,0x05 ,0x15 ,0x00 ,0x04 ,0x00 ,0x00 ,0x00 ,0x3F ,0x04 ,0x00 ,0x00 ,0x00 ,0x3F ,0x00 ,0x00 ,0x00 ,0x00 ,0x40};

void setup() {
    Serial.begin(9600);
    for (byte n = 0; n < 27; n++) {
        Serial.write(dataToSend[n]);
        //~ Serial.print(dataToSend[n], HEX);
        //~ Serial.print(' ');
    }
}

void loop() {
    
}

Program for Receiving on a Mega

// adapted from Serial Input Basics
// char data type replaced by byte
const byte numBytes = 32;
byte receivedBytes[numBytes];
byte numReceived = 0;

boolean newData = false;

void setup() {
    Serial.begin(9600);
    Serial.println("<Arduino is ready>");
    Serial1.begin(9600);
}

void loop() {
    recvBytesWithStartEndMarkers();
    showNewData();
}

void recvBytesWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    byte startMarker = 0x80;
    byte endMarker = 0x40;
    byte rb;
    

    while (Serial1.available() > 0 && newData == false) {
        rb = Serial1.read();

        if (recvInProgress == true) {
            if (rb != endMarker) {
                receivedBytes[ndx] = rb;
                ndx++;
                if (ndx >= numBytes) {
                    ndx = numBytes - 1;
                }
            }
            else {
                receivedBytes[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                numReceived = ndx;  // save the number for use when printing
                ndx = 0;
                newData = true;
            }
        }

        else if (rb == startMarker) {
            recvInProgress = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        for (byte n = 0; n < numReceived; n++) {
            Serial.print(receivedBytes[n], HEX);
            Serial.print(' ');
        }
        Serial.println();
        showGroupsOfBytes();
        newData = false;
    }
}

void showGroupsOfBytes() {
    for (byte n = 0; n < numReceived; n++) {
        Serial.print(receivedBytes[n], HEX);
        Serial.print(' ');
        if ((n + 1) % 5 == 0) {
            Serial.println();
        }
    }
    Serial.println();
}

Hope this helps.

...R

Well gents, after reading through the provided programs and learning some bitwise conversions (left shift and OR), I got my program running and exporting the exact data I wanted to the new displays.

Down the road, there may be some updates for the code to handle new sensors, but for now it reads the sensors available and outputs the data as fast as can be desired. Pretty nice to sit back and see success like this, and thanks all for the support to work through each step of the program, and teaching me other things along the way.

Could you share your code please . I cannot get it to work myself .

boletz:
Could you share your code please . I cannot get it to work myself .

This Thread has been dead for 4 years.

I suggest you start your own Thread and in it post the program you are having trouble with and tell us exactly what the problem is.

...R