Receiving Big Endiang HEX string stream, and parsing to variables

hi guys..
i have been searching and trying bits of code for a while without much luck..

I'm trying to receive HEX strings (stream) from an RS232 serial device
its sent in INT16 - Big Endian and I need to parse the values into variables for use.

the HEX strings are formatted as follows

16 cf 00 00 00 00 00 00 01 4d 00 00 00 18 00 59 0c b4 05 8d 00 01 00 00 00 00 00 04 00 03 00 02 90 04

16 cf is the same start and always ends with the 04

how would I go about reading this from the serial port, convert DEC
and save into variables as this int or float?

var HEX DEC
A 16 CF 5839
B 00 00 0
C 00 00 0
D 00 00 0
E 01 4D 333
F 00 00 0
G 00 18 24
H 00 59 89
I 0C B4 3252
J 05 8D 1421
K 00 01 1
L 00 00 0
M 00 00 0
N 00 04 4
O 00 03 3
P 00 02 2
Q 90 04 -28668

any help / pointers would be greatly appreciated

Rune

What is actually being sent over the serial port?

Does the transmission really consist of the ASCII characters for "1", "6", " ", "C", "F", or "c", "f" etc. (which seems very unlikely) or are you simply representing the binary data using HEX format printout?

thanks for the Responce.

the values i used above are the HEX representation. the ASCII string would resemble this

.Ï.......‹.....a.T.Ž............Ò.

16 cf 00 00 00 00 00 00 01 8b 00 00 00 1a 00 61 .Ï.......‹.....a
0c 60 05 8d 00 01 00 00 00 00 00 04 00 03 00 02 .`.............
de 04 Þ.

the hex values converted to little endian i believe ? is the int16 values i need?
im not entirely sure on the endian part though.

for instance, from the sentence above
01 8B = 395
0C 60 = 3168
05 8D = 1421

i need the values between " 16 cf" and "xx 04" split into variables for use.
continuously updated as its a complete string in "bursts" if you will with the 16 cf being the beginning.

Rune

Those are binary values.

Read them in as a byte or char value and assemble two such in the correct order using shift and bitwise or.

e.g.

unsigned char x = Serial.read();
unsigned char y = Serial.read();
int value = ((x<<8) | y);

Thanks again. for the quick response.

so I guess i would need to read in the whole serial sequence into variables (binary) (one byte per variable).
starting after my initial two bytes (16 cf),

then take a set of two bytes., left shift one 8 bit, and add them up. generating the int16 value output?
repeat for each set?

Rune

I would read in an entire array, up to the terminating 0x04, then process each pair as described.
Something like the following might be a start (untested), but study Serial Input Basics for more info.

//global variables
char buf[40];
int vars[18];
int index=0;

void loop() {
if (Serial.available>=1) {
   buf[index]=Serial.read();
   if (buf[index] == 0x04 && buf[index-1] == 0x90) process_data (); //end of input?
   index++;
   if(index>39) index=0; //out of bounds. input error?
  }
}

void process_data() {
int i=0,j=0, max;
max = index-2;
for (i=2; i<max; i=i+2) { //ignore start and end markers
   vars[j++] = (buf[index]<<8) | buf[index+1];
}
index = 0; //reset global variable index to read in next buffer
}

Note: this simple example assumes that 0x90 0x04 is always an end marker and never appears in the data, which may be a bad assumption. It may be better to count the bytes.

Let us know how it goes!

looks promising. but i think a startmarker is a better way here. as unfortunately only the 04 is static at the end. xx 04

but the startMarker of 16 cf should be static as far as i can tell from the data samples i have

Rune

You may need both the end and start markers. Look at lots of examples to understand what the pattern is.

Are you sure that all the variables are 16 bit ints?

If xx 04 is always at the end, xx might be a useful checksum or CRC character.

it would have been nice with a 2 byte end marker aswell, but i suppose only last 04 isn't much of an end marker.. as my data can certainly have that anywhere.
so its probably going to be start marker bytes + 32 bytes known lenght with the last being the xx 04?

i suppose i could use the 16 cf as end marker. and just read in the next set of bytes.
although probably not the proper way to go about it

i have a pretty good idea of the data structure itself.. and in between the
16 cf and xx 04 i have 15x 16bit int variables.
and yes. the xx does vary with varying data. so i'm sure its a checksum of sorts aswell.

Rune

so its probably going to be start marker bytes + 32 bytes known lenght with the last being the xx 04?

That would be a good starting point.

i think got some material to work with here :o)

thanks for the input

Rune

As i don't have access to the hardware at them moment, I'm using this bit of code on an arduino to generate the pattern. 19200 8N1 continous stream, no newlines etc. just back to back

byte data_message[] = {0x16, 0xcf, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x77, 0x00, 0x00, 0x00, 0x19, 0x00, 0x61, 
                       0x0c, 0x6b, 0x05, 0x8b, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x00, 0x03, 0x00, 0x02, 
                       0xd2, 0x04 };      
void setup() {
  Serial.begin(19200);
}
void loop() {
Serial.write(data_message, sizeof(data_message));  
delay(500);
}

I'm working with Robin2 's excellent examples, using "recvBytesWithStartEndMarkers();"
to get my framing right. as the 0x16, 0xcf bytes I'm using as startmarkers could potentially show up in the data too.

but cant quite get my head around triggering on the right sequence of TWO bytes 0x16, 0xcf, as startmarkers AND a known length, AND with the last byte being 0x04 endmarker

:o

Rune

Fair success now..
I gave up on the "end marker" im just going by start bytes and length.
and it does sort proper into the array and I can shift and parse into variables

(just the two here for simplicity)

however.. consistently every 8 loops. the Arduino resets back to "startup"
i must be doing some buffer overrun or something? cant seem to hunt it down?

Arduino hardware serial output

1527<Arduino is ready>
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
15279317744019061C6B58B015566040302D24
 var1 = 261
 var2 = 519
1527<Arduino is ready>

my Arduino code

// Example to receive data with a start marker and length byte
//   not tested


const byte numBytes = 32; //32
byte receivedBytes[numBytes];
const byte typeBytePosition = 1;//1 // 2nd position after the start byte
const byte requestType = 0x1;//01 0x65;
const byte responseType = 0x1;//01  0x75;
const byte requestLength = 34; //34
const byte responseLength = 34;//34;
static byte numBytesReceived = 0;
boolean newData = false;
#include <SoftwareSerial.h>

SoftwareSerial mySerial(10, 11); // RX, TX

void setup() {
    Serial.begin(19200);
    mySerial.begin(19200);
  Serial.println("<Arduino is ready>");
}

void loop() {
    
    recvBytesWithStartMarker();
    showNewData();
 //   emptyBuffer();
    
}

void recvBytesWithStartMarker() {
  
    static boolean recvInProgress = false;
    static int ndx = 0;
    //static byte numBytesReceived = 0;
    static byte mesgLen = 0;//34
    const byte startMarker = 0x16CF;
    byte rc;

    while (mySerial.available() > 0 && newData == false) {
        rc = mySerial.read();
        numBytesReceived ++;

        if (recvInProgress == true) {
            if (numBytesReceived == typeBytePosition) {
                if (rc == requestType) {
                    mesgLen = requestLength;
                }
                if (rc == responseType) {
                    mesgLen = responseLength;
                }
                ndx = 0; // enable saving of data
            }
            if (ndx >= 0) {
                receivedBytes[ndx] = rc;
                ndx++;
            }
            if (ndx >= mesgLen) { // got the whole message
                receivedBytes[ndx] = 0;
                newData = true;
            }
        }

        else if (rc == startMarker) {
            recvInProgress = true;
            numBytesReceived = 0;
            ndx = -1; // prevent saving data for the moment
        }
    }
}

void showNewData() {
    if (newData == true) {//|| numBytesReceived == numBytes) {
    //    Serial.print("This just in ... ");
        for (byte n = 0; n < numBytes; n++) {
            Serial.print(receivedBytes[n],HEX);
           // Serial.print(' ');
        }
        Serial.println();

int var1 = ((receivedBytes[0]<<8) | receivedBytes[1]);
int var2= ((receivedBytes[2]<<8) | receivedBytes[3]);

Serial.print(" var1 = ");Serial.println(var1);
Serial.print(" var2 = ");Serial.println(var2);
Serial.println();/*


newData = false;
    }
}

im not half sure what half the constants are for. mostly trail and error at this point

Thoughts?

Rune

You MUST check that ndx is never greater than 31 before doing things like this:

               receivedBytes[ndx] = 0;

im not half sure what half the constants are for.

It is very important that you do understand them.

It might be easier to use a state machine to read in the data. You have three states: looking for the start sequence, reading the data, and processing the end marker(s) and the data. Here is some code I wrote for a similar problem, where the 134 byte data record begins with three asterisks and ended with a checksum:

// read bytes from serial port and break out records

    for (i=0; i< n; i++)  //dig through n byte buffer obtained from serial port read routine.
      {
        nchar_total++;  //count all characters received from port

        switch (state) {

        case SEARCH:
            record[index]=buf[i]; //next incoming byte to record
            if(record[index] == '*') { // start sequence?
               if(index > 1 && record[index-1]=='*' && record[index-2]=='*') state=GRAB;
               }
            index++; //continue reading

            if(state == GRAB) { //we have switched states. Clean up.
                    index=0; //reset output record index, don't store '*'
                    nchar_processed += 3; //but count those characters as processed
            }
            break;

        case GRAB:
            record[index]=buf[i];
            nchar_processed++;
            if(index == 130) { // 0 to 130 (131 total) expected, not counting 3x '*'
                process(record); //record complete, process it
                index = 0;
                state = SEARCH; //start looking for "***" sequence again.
                break;
            }
            else index++; //continue storing characters in record buffer
         }