Go Down

Topic: How to best read in Serial packet, parse out several data, rinse, repeat? (Read 15327 times) previous topic - next topic

Robin2

I can't open your file capture_ascii.txt and I can't make sense of capture_HEX.txt.

Can you just extract a single message from the file and post that.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Flinkly

Yeah,  so for the HEX output, "80" is the start byte (Decimal - 128, Binary - 10000000) and "40" is the stop byte (Decimal - 64, Binary - 01000000).  So a packet in HEX is:

[ 800000000220000300051500040000003F040000003F0000000040 ]


EDIT - Actually, you're right, the HEX output doesn't make any sense.  Too many bytes of data for what a packet should be.


EDIT 2 - ACTUALLY, there is a few things going on here that I wasn't expecting, but I believe it's all correct (and kind of makes sense).

First, I only have two sensors, but I have two displays from the same company as well, and it appears that they are submitting their own packet of information to tell other displays down the line that they have already displayed the data for one of the sensors (so you could have two of the same sensor, and two displays showing that KIND of data, and they wouldn't mix the data up). So, a packet for a single sensor/display is 5 bytes, and we're receiving 5 sets of 5 bytes (2 sensors, 2 displays, and see next paragraph...).

Second, the extra data packet is because my second sensor captures vacuum AND pressure data, so it is (apparently) special and sends two sets of data, one for the vacuum side of the scale, the other for the pressure side of the scale (one or the other sends zero data, depending on what the sensor is reading).  so that is where the "extra" set of data is coming from.

So while a lot of this isn't described in the protocol document I have (which isn't really that helpful), I think I get it.


The sensor with the MSB/LSB address of:

00 00
-is an oxygen sensor

00 03
-is a vacuum sensor

00 04
-is a pressure sensor (it's the same physical sensor as the vacuum sensor)

3F 04
-is the display showing the vacuum/pressure sensor data (this section of data might change between 3F 03 and 3F 04 depending on whether the sensor is reading Vacuum or Pressure at that moment)

3F 00
-is the display showing the oxygen sensor data


So overall, this program is going to be a little more work than I imagined.  Also, I'm making this program to replace the displays that I currently have, so while this data is accepting a packet with 5 sets of data in it, I'll only be using it to collect 3 sets of data.  I can collect data without the displays in-line, and would have if I had known they were adding their own information. 

Either way, can still collect this data in the array and just parse out the desired information regardless of what other "data" is coming in the packet.

Robin2

Sorry, but your description of different displays and whatnot is just confusing me.

Let's try to do one part at a time.

Before I do any programming can you confirm that this is a typical message that you need to receive.
        800000000220000300051500040000003F040000003F0000000040
and there are no characters before or after that stream of hex data - such as a CR or LF

And there is something that you have not yet clarified - is that a series of characters 8 0 0 0 etc or is it the HEX representation of the bytes 128 0 0 0 2 32 etc

It would also be useful if you post a version of that message with spaces in it to signify the different groups of bytes that need to be extracted.

When we can receive and parse that data we can consider other issues.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Flinkly

This is a HEX representation of the Bytes of a typical packet, and there aren't any LF or CR.  Here is the data packet with spaces inserted between the HEX numbers:

  80 00 00 00 02 20 00 03 00 05 15 00 04 00 00 00 3F 04 00 00 00 3F 00 00 00 00 40

While a whole packet of typical data is 27 bytes (25 information bytes and a start and stop byte), WITHIN the packet are groupings of 5 bytes (shown below):


  80
  00 00 00 02 20
  00 03 00 05 15
  00 04 00 00 00
  3F 04 00 00 00
  3F 00 00 00 00
  40


Within each grouping of 5 bytes of data, 4 bytes are important to capture for my use, the first two and the last two bytes.

The first two bytes of a grouping of 5 bytes tell me what kind of data is coming.

The last two bytes of a grouping of 5 bytes are the actual raw data.


Does that explain the packet well enough?

Robin2

Does that explain the packet well enough?
Thanks. That seems OK.
I will have a look at it in the morning.
...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Whandall

I have a present for you  ;)

This should handle the framing and decoding.

Code: [Select]

#define TESTING_   // comment for real test

#ifdef TESTING_
const byte testData[] = {
  0x1F, 0x00, 0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00,
  0x00, 0x3F, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x00,
  0x00, 0x40, 0x80, 0x00, 0x00, 0x00, 0x02, 0x20, 0x00, 0x03,
  0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x04,
  0x00, 0x00, 0x00, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x40, 0x80,
  0x00, 0x00, 0x00, 0x02, 0x20, 0x00, 0x03, 0x00, 0x05, 0x15,
  0x00, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x04, 0x00, 0x00, 0x00,
  0x3F, 0x00, 0x00, 0x00, 0x00, 0x40, 0x80, 0x00, 0x00, 0x00,
  0x02, 0x20, 0x00, 0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00,
  0x00, 0x00, 0x3F, 0x04, 0x00, 0x00, 0x00, 0x3F, 0x00, 0x00,
  0x00, 0x00, 0x40, 0x80, 0x00, 0x00, 0x00, 0x02, 0x1F, 0x00,
  0x03, 0x00, 0x05, 0x15, 0x00, 0x04, 0x00, 0x00, 0x00, 0x3F,
  0x04, 0x00, 0x00, 0x00
};

byte testByte;
#endif

const byte startByte = 0x80;
const byte endByte = 0x40;

bool newPacket;

byte packet[25];
byte contains;
bool started;

#ifdef TESTING_
int test_available() {
  return sizeof(testData) - testByte;
}

byte test_read() {
  if (testByte < sizeof(testData)) {
    return testData[testByte++];
  } else {
    return 0;
  }
}
#endif

void processPacket();
void processSerial();

void setup() {
  Serial.begin(115200);
}

void loop() {
  if (newPacket) {
    processPacket();
    newPacket = false;
  }
#ifdef TESTING_
  if (test_available()) {
#else
  if (Serial.available()) {
#endif
    processSerial();
  }
}

int twelveBits(byte* pMsbLsb) {
  return ((pMsbLsb[0] & 0x3F) << 6) | (pMsbLsb[1] & 0x3F);
}

void processPacket() {
  Serial.println("Packet");
  for (byte sensor = 0; sensor < 25; sensor += 5) {
    int adr = twelveBits(&packet[sensor]);
    int data = twelveBits(&packet[sensor + 3]);
    Serial.print("Adr 0x"); Serial.print(adr, 16);
    Serial.print(" Inst "); Serial.print(packet[sensor + 2]);
    Serial.print(" Data 0x"); Serial.println(data, 16);
  }
}

void processSerial() {
  do {
#ifdef TESTING_
    byte newByte = test_read();
#else
    byte newByte = Serial.read();
#endif
    if (!started) {
      started = (newByte == startByte);
    } else {
      switch (newByte) {
        case startByte:
          break;
        case endByte:
          newPacket = (contains == 25);
          started = false;
          break;
        default:
          if (contains < 25) {
            packet[contains++] = newByte;
            continue;
          }
          started = false;
      }
      contains = 0;
      break;
    }
#ifdef TESTING_
  } while (test_available());
#else
  } while (Serial.available());
#endif
}


Result (from embedded testdata):

Code: [Select]
Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0
Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0
Packet
Adr 0x0 Inst 0 Data 0xA0
Adr 0x3 Inst 0 Data 0x155
Adr 0x4 Inst 0 Data 0x0
Adr 0xFC4 Inst 0 Data 0x0
Adr 0xFC0 Inst 0 Data 0x0
Ah, this is obviously some strange usage of the word 'safe' that I wasn't previously aware of. (D.Adams)

Robin2

I have written a short program for an Uno to transmit the data

And I have made a few changes to the example "recvWithStartEndMarkers" from Serial Input Basics so that it works with bytes rather than chars. I also added an extra function so that it will display the data in groups of 5 as well as on one line.

For simplicity I have used a Mega to receive the data as I can use Serial1 and still display the data on the Serial Monitor.

The connections between the two Arduinos are GND to GND and Uno Tx to Mega Rx1.

Uno transmit program
Code: [Select]
byte dataToSend[] = {0x80 ,0x00 ,0x00 ,0x00 ,0x02 ,0x20 ,0x00 ,0x03 ,0x00 ,0x05 ,0x15 ,0x00 ,0x04 ,0x00 ,0x00 ,0x00 ,0x3F ,0x04 ,0x00 ,0x00 ,0x00 ,0x3F ,0x00 ,0x00 ,0x00 ,0x00 ,0x40};

void setup() {
    Serial.begin(9600);
    for (byte n = 0; n < 27; n++) {
        Serial.write(dataToSend[n]);
        //~ Serial.print(dataToSend[n], HEX);
        //~ Serial.print(' ');
    }
}

void loop() {
   
}


Program for Receiving on a Mega
Code: [Select]
// adapted from Serial Input Basics
// char data type replaced by byte
const byte numBytes = 32;
byte receivedBytes[numBytes];
byte numReceived = 0;

boolean newData = false;

void setup() {
    Serial.begin(9600);
    Serial.println("<Arduino is ready>");
    Serial1.begin(9600);
}

void loop() {
    recvBytesWithStartEndMarkers();
    showNewData();
}

void recvBytesWithStartEndMarkers() {
    static boolean recvInProgress = false;
    static byte ndx = 0;
    byte startMarker = 0x80;
    byte endMarker = 0x40;
    byte rb;
   

    while (Serial1.available() > 0 && newData == false) {
        rb = Serial1.read();

        if (recvInProgress == true) {
            if (rb != endMarker) {
                receivedBytes[ndx] = rb;
                ndx++;
                if (ndx >= numBytes) {
                    ndx = numBytes - 1;
                }
            }
            else {
                receivedBytes[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                numReceived = ndx;  // save the number for use when printing
                ndx = 0;
                newData = true;
            }
        }

        else if (rb == startMarker) {
            recvInProgress = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        for (byte n = 0; n < numReceived; n++) {
            Serial.print(receivedBytes[n], HEX);
            Serial.print(' ');
        }
        Serial.println();
        showGroupsOfBytes();
        newData = false;
    }
}

void showGroupsOfBytes() {
    for (byte n = 0; n < numReceived; n++) {
        Serial.print(receivedBytes[n], HEX);
        Serial.print(' ');
        if ((n + 1) % 5 == 0) {
            Serial.println();
        }
    }
    Serial.println();
}


Hope this helps.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Flinkly

Well gents, after reading through the provided programs and learning some bitwise conversions (left shift and OR), I got my program running and exporting the exact data I wanted to the new displays.

Down the road, there may be some updates for the code to handle new sensors, but for now it reads the sensors available and outputs the data as fast as can be desired.  Pretty nice to sit back and see success like this, and thanks all for the support to work through each step of the program, and teaching me other things along the way.

boletz

Could you share your code please . I cannot get it to work myself .

Robin2

Could you share your code please . I cannot get it to work myself .
This Thread has been dead for 4 years.

I suggest you start your own Thread and in it post the program you are having trouble with and tell us exactly what the problem is.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Go Up