Help with Serial Hex Data

Hi,

I’m working on reading a 5 byte hex code that’s being sent from a Digi Connectport x4 via serial to an Mega ADK. The data is being sent every 30 seconds. The format of the data is : 0xAA, 0x"", 0x"", 0x"FFFF", with 0xAA as the header, the 2 bytes contain the voltage value from a remote xbee that has been parsed and packed, the 0x"FFFF" is a crc value.

My general input that comes is in 0xAA 0x03, 0xBC, 0xA1, 0x50.

What I’m having trouble with is pulling the packed value out of the data stream. I will occasionally loose some part of the packet and I don’t know how to fix that… and I have trouble with the CRC calculation. I’ve commented out some of the xmodem crc stuff that I’ve seen online for the moment.

Code is as follows:

#include <crc16.h>

byte data[5]; //For holding the ID we receive
int val = 0;
int voltage = 0;

unsigned char mess[5] = {data[5]};
void setup()
{
// Start serial port 19200 bps
uint16_t crc_out; //crc call
Serial.begin(9600); //start serial
Serial1.begin(9600); //start serial1
// crc_out = calc_crc(mess,10,0xFFFF); //calcs crc on the fly
// Serial.println(crc_out, HEX); //expecting 0xFFFF

// delay(500);
// Serial.flush();
// Serial.println();
// Serial.println(“RFID module started in Auto Read Mode, Waiting for Card…”);
}

void loop()
{
val = Serial1.read();
while (val != 0xAA)
{
val = Serial1.read();
delay(2000);
}
//we already read the header (0xAA)
data[0] = Serial1.read(); // Lo-HEX
data[1] = Serial1.read(); // High-Hex
data[2] = Serial1.read(); // checksum
data[3] = Serial1.read(); // checksum
// Hex voltage value
delay(50);
Serial.print("Data Input: ");
for (int i = 0; i < 2; i++)
{
if (data == 4);
_ Serial.print(data*, HEX);_
_
//delay(100);_
_
//Serial.println();_
_
voltage = map(data[0,1], 0, 1023, 0, 255);_
_
//Serial.println();_
_
Serial.print(voltage);_
_
}_
_
Serial.println();_
//crc_out = calc_crc(mess,10,0xFFFF); //calcs crc on the fly*

* // Serial.println(crc_out, HEX); //expecting 0xFFFF*
}
// uint16_t calc_crc(unsigned char msg,int n,uint16_t init)
_
//{_
// uint16_t x = init;
_
// while(n–)_
_
// {_
// x = crc_xmodem_update(x, (uint16_t)msg++);
_
// }
_
// return(x);
//}
//uint16_t crc_xmodem_update (uint16_t crc, uint8_t data)
//{
// int i;
// crc = crc ^ ((uint16_t)data << 8);
//for (i=0; i<8; i++)
//{
// if (crc & 0x8000)
// crc = (crc << 1) ^ 0x1021; //(polynomial = 0x1021)
//else
// crc <<= 1;
//}
// return crc;
//}
My current output is : 3BC

  1. unsigned char mess[5] = {data[5]}; what do you thing you are doing here!!!!

2.

void loop()
{
  val = Serial1.read();

You have no right to assume that there will be any data available!..

 data[0] = Serial1.read();    // Lo-HEX
  data[1] = Serial1.read();    // High-Hex
  data[2] = Serial1.read();    // checksum
  data[3] = Serial1.read();    // checksum

Again you have no right to assume that ANY data is avaliable let alone all 5 (or is it 6) bytes.

Over all score 0/1000.

Mark

You need a much more structured approach to receiving serial data. The Arduino works a great deal faster than the data can arrive.

You haven't said if there is always a fixed number of bytes and/or whether there is a terminating byte that can be used to tell when all the data has been received.

There are different ways of receiving the data depending on what you know about the incoming data. If there is always a fixed number N of bytes you can use if (Serial.available >= N) { to check if they have all arrived before starting to read them.

Having a terminating byte is probably more reliable if you have that option. The code in Chapter 8 in this Demo shows how to do that.

Another thing that isn't clear from your post is whether your Arduino is receiving alphanumeric characters representing HEX values or just numbers equivalent to those HEX values.

...R

Robin2,

Thanks for the actual help. I’m new to programming and this is a pretty big jump from Matlab which I have used briefly in the past. I know it’s common for people to be dicks in this forum so it’s really nice to be pushed in the right direction by someone who cares about helping those of us who aren’t experienced with programming.

The data is coming in with a fixed number of bytes, 5. It is using 2 crc16 end bytes,( 0xFFFF is pulled with the calc from a python program) - to terminate the message. Which is going to change with each message based on the pulled values on the other end.

I’ve done as you suggested and included the if(Serial.available >=5) {…
This has I think instructed the program to wait until all 5 bytes are received.

The end goal of this program is to check to see if the header has been received, then take the 2 following bytes and convert them to a PWM output. Assuming that the CRC code checks out.

This is also the 1st program I’ve written for arduino so I’m assuming there will be a huge learning curve here.

#include <util/crc16.h>

byte data[5];
int val = 0;

void setup()
{
Serial.begin(9600); //start serial
Serial1.begin(9600); //start serial1
}

int voltage = 0;
void loop()
{
if (Serial1.available() == 5) {
int inByte = Serial1.read();
//Serial.println(inByte);

uint16_t crc = 0xFFFF;
uint8_t data = {‘inByte’};
uint8_t i;

for (i = 0; i < 5; i++)
crc = _crc16_update(crc, data*);*

  • Serial.println();*

  • Serial.println(crc, HEX);*

  • delay(100);*

  • val = (inByte);*

  • while (val != 0xAA)*

  • {*

  • val = Serial1.read();*

  • delay(2000);*

  • }*

  • //we already read the header (0xAA)*

  • data[0] = Serial1.read(); // Lo-HEX*

  • data[1] = Serial1.read(); // High-Hex*

  • data[2] = Serial1.read(); // checksum*

  • data[3] = Serial1.read(); // checksum*

  • Serial.print(data[0], HEX);*

  • Serial.print(data[1],HEX);*

  • Serial.println();*

  • // Hex voltage value*

  • delay(50);*

  • Serial.print("Voltage in Hex: ");*

  • for (int i = 0; i < 2; i++)*

  • {*
    _ if (data == 4);_
    _ Serial.print(data*, HEX);
    delay(100);
    // voltage = map(data, 0, 1023, 0, 1023);
    // Serial.print(voltage);
    // Serial.println();
    }
    Serial.println();
    Serial.println(“CRC Hex”);
    Serial.print(data[2], HEX);
    Serial.print(data[3], HEX);
    //int CRC = {(data[2] + data[3])};
    //Serial.print(CRC, HEX);*_

* Serial.flush();*
* Serial1.flush();*
* Serial.println();*
* }*
}
[/quote]
Produces :
> 3F47
> 0C
> Voltage in Hex: 0C
> CRC Hex
> E551
thanks for the help

  1.   if (Serial1.available() == 5) { it may never be equal to 5 therefore the test is >= not ==

  2.     int inByte = Serial1.read(); read returns 1 byte. using an arrays name just gives you the address of the first element in the array

  3. val = (inByte); Dito

    while (val != 0xAA)
    {
      val = Serial1.read();
      delay(2000);
    }

WTF read data until the next message at a rate of 1 char/(2 seconds) and throw it all away.

5 Again POST CODE IN CODE TAGS.

I suggest you start by reading the two stickys at the top of the forum. Them the reference pages and follow the links. questions will be asked.

Mark

There are still a number of errors in your program. I was not able to get it to compile because I don't have your include file. Still, even if I did, I think there would still be a number of issues. To get you a little more comfortable with the Serial object, try the following simple program:

void setup() {
  Serial.begin(9600);
}

int counter = 0;
byte data[15];
void loop() {

  if (Serial.available()) {
    data[counter] = Serial.read();
    Serial.println(data[counter++]); 
  }

}

First, don't type anything but click the Send button on the monitor. The last number you see is a 10, which is the Linefeed character ('\n'). (There may be some garbage before it.) Now type in "123" on the Serial input, click Send, and what do you see? You should see 49, 50, 51. If you look at an ASCII character table, you will see those are the ASCII codes for a '1', '2', and '3'. But also note that you still get a 10 at the end. So, if you want to read data up to when the user clicks the Send button, try:

Serial.readBytesUntil('\n', data, 15);

where the 15 is the max number of bytes you want to read. You should try to work this into your program until you understand what each line is doing. I think that will help you see where your errors are.

fixed #1

in response to #2 & #3 would it be better to do as follows:

 val = (Serial1.read());

Would that be the appropriate way to solve the issue? Or should the data be stored in a buffer and read from there?

Regarding 4,

I'm obviously looking for the header of 0xAA, which comes through with every packet. I don't know why but the delay seems necessary, when it is not included I just get garbage for a return or nothing. When it is set to a minimum of 2 seconds I get the rest of the packets.

I don’t know why but the delay seems necessary,

When you code it correctly it will not be necessary.

I’m obviously looking for the header of 0xAA, which comes through with every packet.

while(Serial.read() != 0xAA) { }

Will wait until the first byte of you packet arrives. If you read an empty buffer you get 0xff so that line will hold until you have aligned your data stream and got the first header byte.
Then you can go on and read the rest of your values as they appear in the input buffer. Do not try and read them until they do appear.
You can do this with a :-

while (Serial.available() < restOfPacket ) { }

Note the { } will do nothing between the tests.

Try this to start with. It should receive all 5 bytes and print them out. When that is working you can add other bits and pieces. (I haven’t tested it).

byte numInput = 5;
byte inData[8];
boolean newData = false;

void setup() {
  Serial.begin(9600);
  Serial1.begin(9600);
}

void loop() {
  readSerialOne();
  displaySerialOne();
  checkCRC();
}

void readSerialOne() {
  if (Serial1.available >= numInput) {
    for (byte n = 0; n < numInput; n ++) {
       inData[n] = Serial1.read();
    }
    newData = true;
  }
}

void displaySerialOne() {
  if (newData == false) {
    return;
  }
  for (byte n = 0; n < numInput; n ++) {
    Serial.print(inData[n], HEX);
    Serial.print("  ");
  }
  Serial.println();
  newData = false;
}

void checkCRC() {
  // to be completed
}

…R

@robin2 I'm assuming that there is no request/response thing going on. So we need to sync up with the first 0xAA. We can't just read them in batches of 5 and assume that each will start with 0xAA.

We could do something in setup to get round this but here is another way.

PEUSDO code warning

byte buf[5];
void loop(){
  if (Serial1.peek()==0xAA){
      //head of message
      if (Serail1.available()>=5){
          // here the first byte will be the message header AND there wiil be a full 5 byte message in serial1's buffer.
          buf[0]=Serial1.read();// read the header
          buf[1]=Serial1.read();// read the first data byte
          buf[2]=Serial1.read();// read the second data byte
          buf[3]=Serial1.read();// read the first crc byte
          buf[4]=Serial1.read();// read the second crc byte
          Serial.println(buf,hyex); // debug print out
          processMessage();// you write this inc crc validation

      }
  } else {
       Serial.read(); // throw away the rubbish
  }
   
}

Mark

PEUSDO code warning

That doesn’t even work as pseudo code.

Get the header byte and then immediately if 5 bytes have not arrived, loop round again and when the next byte arrived it will not be the header and so it is discarded?

Pseudo code should at least have the right structure even if it does not have the right syntax.

peek() not read(). Look in the reference.

See also single char/byte look ahead. But I did re-invent print :)

Mark

holmes4:
peek() not read(). Look in the reference.

I am well aware you used peek, but your logic is wrong because it assumes that all the data is in when you see the header.

No, hence the nested available()>5.

Mark

Yes and if five are not Avaliable where does the program go next?

This may not help at all, but this is how I'd start from reading your question.

/*
index keeps track of the index for the next data byte in your buffer and is also looked at for state.
 0 -> waiting
 1…4 -> getting data
 5 -> buffer full,
 calculate stuff for the rest of the code and reset index to 0
 */

int index;               // Holds where next data byte will go into your buffer.
byte buffer[5];         // Holds the data.


// checkInput function that will keep track of incoming data stream.
boolean checkInput() {

  boolean dataReady = false;    // Used to tell the rest of the program we have something for them..

  if (index>=5) {      // We have a full buffer? Do the math!

    // Calculations       // Doing whatever you need to do to the buffer.

    index = 0;            // Reset the index for the next batch.
    dataReady = true;     // To let the world know.
  } 
  else if (Serial.available()) {                 // Buffer is NOT full. We have something?
    buffer[index] = Serial.read();           // Grab it!
    if ((index==0 && buffer[0]==0xAA) || (index>0)) { // We got the header at the start, -or- a data byte. good!
      index++;                       // In all these cases its time to increment the index.
    }
  }                                    // The other case is data at the start, so we ignore it and wait for a header.
  return(dataReady);                   // If the data is processed, let 'em know.
}


void setup() {

  index = 0;               // Set the index at the beginning of the buffer.
  
  // other setup stuff         // Other things you'd like to setup.

}


void loop() {

  if (checkInput()) {
      // do what you do if there is data ready.
  }
     // do what you do on every loop() call..
}

If this just makes things more confusing.. Just ignore it.

-jim lee

Back round the loop (and why not!), you don't have to read from serial just because its got data in it!.

We only deal with the message when we have all 5 bytes AND the first byte is the correct header. Previous posts in the thread already assume that we get the message as a "whole" my method ensures this.

Mark

holmes4: @robin2 I'm assuming that there is no request/response thing going on

According to the original post there is a huuuuuugggggge 30 second gap that can be used for synchronizing.

...R