How to best read in Serial packet, parse out several data, rinse, repeat?

So I'm working on reading in from a bank of sensors (set up in serial, no pun intended) that have a specific serial protocol: start, stop, sensor type, sensor instance (for multiple of the same kind), sensor data, etc. The packets come at 100ms intervals, if that helps, and I can't imagine having a packet larger than 26 bytes in a packet (if I remember correctly, I don't have the protocol sheet I front of me at the moment).

I've got my program open to reading the data (pretty easy to Serial.begin() ), and know how to convert the raw data into usable information and then display it on my 7 segment(s), but I'm at a loss at how to somehow read in a whole packet, and dissect it for the pertinent data.

I've read through the Serial commands, but since I don't really program, I just can't seem to see the commands I need to make this happen.

I can add in my program so far, but at the moment I'm just reading the bytes available in the buffer to make sure I am properly connected, then the giant section where I grab and dissect data (missing - I'm here for guidance/assistance), then I have simple conversion and display (currently displaying the full buffer number - 64).

Does the device send an end of data packet marker along with the data? Below is very simple serial capture code.

//zoomkat 6-29-14 Simple serial echo test
//type or paste text in serial monitor and send

String readString;

void setup() {
  Serial.begin(9600);
  Serial.println("Simple serial echo test"); // so I can keep track of what is loaded
}

void loop() {

  while (Serial.available()) {
    char c = Serial.read();  //gets one byte from serial buffer
    readString += c; //makes the String readString
    delay(2);  //slow looping to allow buffer to fill with next character
  }

  if (readString.length() >0) {
    Serial.println(readString);  //so you can see the captured String 
    readString="";
  } 
}

One of the examples in Serial Input Basics should do what you want.

They read in all the data before trying to figure out what it contains. There is also a parse demo.

...R

Thank you two for the quick responses.

I think I'm going to need to take this program in baby steps, because while I believe I understand most of what the provided code is doing, when applying it to my situation it quickly becomes overwhelming.

So I have 2 sensors at the moment, so I know what my packet should look like:

Start byte

• 1000 0000

Sensor n Address MSB byte
Sensor n Address LSB byte

Sensor n Instance byte (if I was using more than 1 of a single sensor type at once)

Sensor n Data MSB byte
Sensor n Data LSB byte

Sensor n+1 Address MSB byte
Sensor n+1 Address LSB

Sensor n+1 Instance byte

Sensor n+1 Data MSB byte
Sensor n+1 Data LSB byte

Stop Byte

• 0100 0000

So I need to read in the packet, separate the bytes, read sensor n address, manipulate sensor n data, read sensor n+1 address, manipulate sensor n+1 data, send to displays, rinse and repeat.

For starters, while I can read this packet and bytes in as whatever I want, I will want them as integers for both the sensor address and sensor data, so instead of "char" or "byte", I should read the packet and individual bytes in as "int", correct? (baby steps - I'm an ME by degree, so bear with me)

Second, I think reading in 1 complete packet and displaying that on the serial monitor is a good starting point, so that's what I've done in the code below, which will let me see how each byte is separated (I believe it's by spaces, as opposed to commas. The paperwork I have states that they used standard RS232 UART serial protocol, if that means anything to y'all - it doesn't to me.)

const byte numInts = 32;
int receivedInts[numInts];

boolean newData = false;


void setup() {                 // Put your setup code here, to run once

  Serial.begin(19200);         // Start serial connection
  while (!Serial);             // Not really sure what this is for, but examples have it...

}


void loop() {                  // put your main code here, to run repeatedly
  
  recvWithStartEndInt();       // ...
  showNewData();               // ...

}


void recvWithStartEndInt() {
  
  static boolean recvInProgress = false;
  static byte ndx = 0;         //
  int startInt = 128;          // 128 is integer equivalent of 1000 0000 <- start bit
  int endInt = 64;             // 64 is integer equivalent of 0100 0000 <- stop bit
  int ri;                      //
  
  while (Serial.available() > 0 && newData == false) {
   ri = Serial.read();

   if (recvInProgress == true) {
     if (ri != endInt) {
       receivedInts[ndx] = ri;
       ndx++;
       if (ndx >= numInts) {
         ndx = numInts - 1;
       }
     }
     else {
       receivedInts[ndx] = '\0'; // terminate the string
       recvInProgress = false;
       ndx = 0;
       newData = true;
     }
    }

    else if (ri == startInt) {
      recvInProgress = true;
    }
  }
  
}


void showNewData() {
  if (newData == true) {
    Serial.print ("This just in ...");
    Serial.println(receivedInts);
    newData = false;
  }
}

...but it won't upload because:

no known conversion for argument 1 from 'int [32]' to 'long unsigned int'
call of overload 'println(int [32])' is ambiguous

There is one other thing that confuses me, the MSB and LSB callouts for the packet information. When looking at the raw data in the packet, it appears to me that I can only get 0 to 63 as either my MSB or LSB sensor data, but the conversion from raw to real data requires 0-255 as an input to get the entire scale of the sensor...

I must be missing something.

The protocol documentation does state that ONLY the start byte will use the first bit in any byte, and ONLY the stop byte will use the second bit in any byte.

So in other words, ALL other bytes in the packet will have zero's for the first two bits.

It also goes on to state that:

To interpret the Sensor Address and Data Value a bitwise conversion must be done.

While I'm a little lost as to what this means for my program, I guess my assumption in the previous post about using "int" is incorrect. It sounds like I need to use "byte" and then convert the bytes into usable data...

Flinkly:
Second, I think reading in 1 complete packet and displaying that on the serial monitor is a good starting point, so that's what I've done in the code below, which will let me see how each byte is separated (I believe it's by spaces, as opposed to commas. The paperwork I have states that they used standard RS232 UART serial protocol, if that means anything to y'all - it doesn't to me.)

You are getting mixed up between bytes (or chars) and ints.

Serial.read() reads a byte (because that is how serial data is sent). So go back to my original version of the code and, just change char to byte.

Then see what you are receiving and we will try to take it a step forward.

...R

Newly minted program.

And this time I read ALL the notes on the program and got past my compile issue caused by println (changed println(recievedBytes) to println((char*)recievedBytes).

const byte numBytes = 32;
byte receivedBytes[numBytes];

boolean newData = false;


void setup() {                              // Put your setup code here, to run once

  Serial.begin(19200);                      // Start serial connection
  while (!Serial);                          // Not really sure what this is for, but examples have it...

}


void loop() {                               // put your main code here, to run repeatedly
  
  recvWithStartEndBytes();                  // ...
  showNewData();                            // ...

}


void recvWithStartEndBytes() {
  
  static boolean recvInProgress = false;
  static byte ndx = 0;                      //
  byte startByte = 10000000;                // <- start byte
  byte endByte = 01000000;                  // <- stop byte
  byte rb;                                  //
  
  while (Serial.available() > 0 && newData == false) {
   rb = Serial.read();

   if (recvInProgress == true) {
     if (rb != endByte) {
       receivedBytes[ndx] = rb;
       ndx++;
       if (ndx >= numBytes) {
         ndx = numBytes - 1;
       }
     }
     else {
       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;
       ndx = 0;
       newData = true;
     }
    }

    else if (rb == startByte) {
      recvInProgress = true;
    }
  }
  
}


void showNewData() {
  if (newData == true) {
    Serial.print ("This just in ...");
    Serial.println((char*)receivedBytes);
    newData = false;
  }
}

Ran it off of the serial transmission from my sensors and on the serial monitor all I got was line after line of:

This just in...

When I unplugged the serial Rx, the arduino would stop sending "This just in...", but with it plugged in, I didn't seem to get the additional "recievedBytes" packet printed as I expected to.

These are wrong
byte startByte = 10000000; // <- start byte
byte endByte = 01000000; // <- stop byte
They should be
byte startByte = 0b10000000; // <- start byte
byte endByte = 0b01000000; // <- stop byte
to tell the compiler it is binary data

Alternatively just use
byte startByte = 128; // <- start byte
byte endByte = 64; // <- stop byte

And modify this piece of code to show how many bytes are received each time

       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;
       Serial.print("Num Bytes ");    // <---- new
       Serial.println(ndx);                 // <---- new
       ndx = 0;
       newData = true;

...R

I was just about to try:

byte startByte = B10000000;

after reading this:

https://www.arduino.cc/en/Reference/Byte

but I went ahead and used "128" and "64" instead.

As far as the new code/program output, now I get this over and over:

This just in ...
Num Bytes 5

The fact that it's sending and re-sending both these lines, as well as showing a numBytes not equal to 32 means that something's being captured, just not printed back over serial.

While it would be nice to see what the arduino sees for a packet, I think I'll try to parse the individual bytes from the packet and see what I can do/make with that and print back over serial.

You need to give me an example of a complete packet as it is transmitted. Then I can do some experiments myself.

If you are expecting to receive 32 bytes you should increase the receive array to (say) 35 bytes to make sure there is no chance of overflow or missing something.

...R

Haven't had time to work on it today (I goofed on a circuit design and am scrambling to make the best of a very expensive mistake), BUT I do have a TXT file from Putty of some output from a few months back. It's in ASCII, but with some work can be translated to whatever.

Let me know if you'd like it attached until I can get my Arduino to output something.

And thank you for all the help.

Or if there is some way to accept in serial over the Rx pin and output it directly to the serial monitor, I can provide output from there as well.

Flinkly:
BUT I do have a TXT file from Putty of some output from a few months back. It's in ASCII, but with some work can be translated to whatever.

Three points ...
If the file contains the stuff that the Arduino program is expected to receive it is what I want. But it would be important to identify what is considered to be a single message so I can program an Uno to send it.

When you say it is in ASCII it make me wonder if that is the format that the Arduino will be receiving? I need to be able to emulate the exact data that your Arduino will be receiving. If you just mean that it is the ASCII values of characters (65 instead of 'A') that is fine.

If there are non-printing characters (such as CR and LF) in the data I need to be able to see them.

Attaching a text file will be fine - hope it is not too long.

...R

So I'm a little proud of myself, because I got output finally.

I tried a few things like "Serial.write()", but I finally just added a "Serial.println(receivedBytes[ndx], BIN)" as the program placed each byte received over serial into the receivedBytes[] array.

From this I received:

0
0
0
1
110111

The first three bytes are expected to be zero for this sensor setup, and the last two bytes are the data. I'm slightly bothered that the output doesn't show all 8 "digits" of the full bytes, but I'm sure that's something I can tell the Arduino to do if I knew how.

Again, here is the protocol of the input I'm EXPECTING to be receiving:

Sensor Address MSB
Sensor Address LSB // <- the sensor connected has an address value of zero
Sensor Instance // <- only one of this type of sensor is connected, so this is again zero
Sensor Data MSB // Now I just need to figure out what bitwise conversion is needed for real data
Sensor Data LSB //

I think I'll connect my second sensor now so that I can make my program able to differentiate between different sensors and apply conversions to the data as necessary. I think I am starting to get the hang of this program now.

Code attached for completeness of post:

//#include <Wire.h>

//#include "Adafruit_LEDBackpack.h"
//#include "Adafruit_GFX.h"

const byte numBytes = 32;
byte receivedBytes[numBytes];

boolean newData = false;

//Adafruit_7segment matrix = Adafruit_7segment();


void setup() {                              // Put your setup code here, to run once

  Serial.begin(19200);                      // Initialize serial connection
  while (!Serial);                          // Not really sure what this is for, but examples have it...

  //matrix.begin(0x70);                       // Initialize 7Segment
  //matrix.print(10000, DEC);                 // Print "----" to 7Segment
  //matrix.writeDisplay();                    // Write the previous Print to the 7Segment

}


void loop() {                               // put your main code here, to run repeatedly
  
  recvWithStartEndBytes();                  // ...
  //showNewData();                            // ...

}


void recvWithStartEndBytes() {
  
  static boolean recvInProgress = false;
  static byte ndx = 0;                      //
  byte startByte = 128;                     // <- start byte
  byte endByte = 64;                        // <- stop byte
  byte rb;                                  //
  
  while (Serial.available() > 0 && newData == false) {
   rb = Serial.read();

   if (recvInProgress == true) {
     if (rb != endByte) {
       receivedBytes[ndx] = rb;
       
       Serial.println(receivedBytes[ndx], BIN);
       
       ndx++;
       if (ndx >= numBytes) {
         ndx = numBytes - 1;
       }
     }
     else {
       receivedBytes[ndx] = '\0';            // terminate the string
       recvInProgress = false;

       Serial.println();                         //adds a carriage return after each complete packet

       //Serial.print("Num Bytes ");
       //Serial.println(ndx);

       //Serial.print("This just in ...");
       //Serial.write(receivedBytes, );
       
       //matrix.println(ndx);
       //matrix.writeDisplay();
       
       ndx = 0;
       newData = true;
     }
    }

    else if (rb == startByte) {
      recvInProgress = true;
    }
  }
  
}


void showNewData() {
  if (newData == true) {
    //Serial.print("This just in ...");
    //Serial.write(receivedBytes, );
    newData = false;
  }
}

Flinkly:
Again, here is the protocol of the input I'm EXPECTING to be receiving:

As I said earlier if you provide a sample of the data I will try some code myself.

...R

Ok, found a few terminal programs and one worked pretty well and I got two outputs (one hex, the other ascii) and a screen capture of binary data.

Also, I connected the second sensor, so this is twice as large of a data packet as I have been receiving so far. So this data would create a 10 item array instead of the 5 item array I have been getting when testing the program earlier.

capture_HEX.txt (9.8 KB)

capture_ascii.txt (21.8 KB)

Screen_binary.jpg937×314 273 KB

I can't open your file capture_ascii.txt and I can't make sense of capture_HEX.txt.

Can you just extract a single message from the file and post that.

...R

Yeah, so for the HEX output, "80" is the start byte (Decimal - 128, Binary - 10000000) and "40" is the stop byte (Decimal - 64, Binary - 01000000). So a packet in HEX is:

[ 800000000220000300051500040000003F040000003F0000000040 ]

EDIT - Actually, you're right, the HEX output doesn't make any sense. Too many bytes of data for what a packet should be.

EDIT 2 - ACTUALLY, there is a few things going on here that I wasn't expecting, but I believe it's all correct (and kind of makes sense).

First, I only have two sensors, but I have two displays from the same company as well, and it appears that they are submitting their own packet of information to tell other displays down the line that they have already displayed the data for one of the sensors (so you could have two of the same sensor, and two displays showing that KIND of data, and they wouldn't mix the data up). So, a packet for a single sensor/display is 5 bytes, and we're receiving 5 sets of 5 bytes (2 sensors, 2 displays, and see next paragraph...).

Second, the extra data packet is because my second sensor captures vacuum AND pressure data, so it is (apparently) special and sends two sets of data, one for the vacuum side of the scale, the other for the pressure side of the scale (one or the other sends zero data, depending on what the sensor is reading). so that is where the "extra" set of data is coming from.

So while a lot of this isn't described in the protocol document I have (which isn't really that helpful), I think I get it.

The sensor with the MSB/LSB address of:

00 00
-is an oxygen sensor

00 03
-is a vacuum sensor

00 04
-is a pressure sensor (it's the same physical sensor as the vacuum sensor)

3F 04
-is the display showing the vacuum/pressure sensor data (this section of data might change between 3F 03 and 3F 04 depending on whether the sensor is reading Vacuum or Pressure at that moment)

3F 00
-is the display showing the oxygen sensor data

So overall, this program is going to be a little more work than I imagined. Also, I'm making this program to replace the displays that I currently have, so while this data is accepting a packet with 5 sets of data in it, I'll only be using it to collect 3 sets of data. I can collect data without the displays in-line, and would have if I had known they were adding their own information.

Either way, can still collect this data in the array and just parse out the desired information regardless of what other "data" is coming in the packet.

Sorry, but your description of different displays and whatnot is just confusing me.

Let's try to do one part at a time.

Before I do any programming can you confirm that this is a typical message that you need to receive.
800000000220000300051500040000003F040000003F0000000040
and there are no characters before or after that stream of hex data - such as a CR or LF

And there is something that you have not yet clarified - is that a series of characters 8 0 0 0 etc or is it the HEX representation of the bytes 128 0 0 0 2 32 etc

It would also be useful if you post a version of that message with spaces in it to signify the different groups of bytes that need to be extracted.

When we can receive and parse that data we can consider other issues.

...R

This is a HEX representation of the Bytes of a typical packet, and there aren't any LF or CR. Here is the data packet with spaces inserted between the HEX numbers:

80 00 00 00 02 20 00 03 00 05 15 00 04 00 00 00 3F 04 00 00 00 3F 00 00 00 00 40

While a whole packet of typical data is 27 bytes (25 information bytes and a start and stop byte), WITHIN the packet are groupings of 5 bytes (shown below):

80
00 00 00 02 20
00 03 00 05 15
00 04 00 00 00
3F 04 00 00 00
3F 00 00 00 00
40

Within each grouping of 5 bytes of data, 4 bytes are important to capture for my use, the first two and the last two bytes.

The first two bytes of a grouping of 5 bytes tell me what kind of data is coming.

The last two bytes of a grouping of 5 bytes are the actual raw data.

Does that explain the packet well enough?

Flinkly:
Does that explain the packet well enough?

Thanks. That seems OK.
I will have a look at it in the morning.
...R