Decoding serial port data

[gerthe]

Hi

I reach out to this forum to see if someone can help me to explain how the Serial.read() function actually works on a Arduino Mega board.

I have read dozens of articles and examples and it seems straight forward and quite simple - but I'm completely unable to to get it to work.

A small background to the project I'm working with.
I'm have a task where i will try to parse a data stream coming from an application called "CondorUDP2Com". The use of this program is to decode a data stream from Condor Soaring 2 and present it on a com port so it can be used for example by Arduinos.
I will use this data to ultimately drive step motors so simulate instruments in a cockpit.

Data will arrive in frames of a header , start of data marker , variable index and a float value.
An example of a data frame is like this.

78 78 78 78 78 78 xxxxxx
7a 01 c5 54 da 41 z 1 27.29139 // Speed in knots
7a 02 f6 66 79 44 z 2 997.60876 // Altitude in feet
7a 03 e2 55 54 bf z 3 -0.829443 // Vario m/s ??
7a 04 f4 55 54 bf z 4 -0.829443 // Electric vario m/s ??

(Info after the 6 HEX values is just my own notes to know what I expect to see at the end)

The programmer of the CondorUDP2Com has described the frame as

At the beginning of each frame, 6 bytes (hex: 78) are sent. Then to indicate which value is sent over, two bytes are transmitted (first is always 7A and second is an integer indicating index from "UDP preview". Then 4 bytes are sent, which is just a float value.

Enough background
I use a Arduino Mega 2560 bord and I have connected Digital pin 18/19 as Serial1, Serial (USB) used for monitoring and upload of data.
If I try to fetch data from the CondorUPD2Com directly or if I try to emulate data by sending it via the programs Termite Terminal or Pololu Serial Transmitter don't I get the data I expect

To my understanding so should Serial1.read() rad one byte , in my world so should it have been one "78" (HEX) corresponding to a char/integer value of 120 in the Arduino.
But if I try so simulate the header by sending 0x78 in so do I get it decoded as

Setup finished
Available bytes: 1
Read hex: 232

1 loop(s) done

Available bytes: 5
Read hex: 232

2 loop(s) done

Available bytes: 4
Read hex: 232

3 loop(s) done

Available bytes: 3
Read hex: 232

4 loop(s) done

Available bytes: 2
Read hex: 232

5 loop(s) done

Available bytes: 1
Read hex: 8

6 loop(s) done

Why do I get 232 (HEX 9A) and not 120 as I expect ? And why is the last byte a 8 and not a 120?

My expectation is to receive 120 120 120 120 120 120 equivalent of x x x x x x as the beginning of the data frame is.

Attached is the program I'm trying with for the moment - disclaimer I'm a newbie programmer when it comes to Arduino code, and it contains a lot of bloat variables that are a result of hours of unsuccessful attempts to identify the incoming data.

The way I try to tackle this problem is to first sync into a data fram by reading the "78":s (x) and discarding them, when a "79" (z) is encountered do I know that I'm on a "data line" - discard that one.
Read the variable and the float - will probably be done via arrays.

But now is the program stripped down to just trying to identify a header sign - and I'm not able to do it.

Thanks a lot for some input as this drives me insane as i can't figure out what is going wrong
Probably some very easy data conversion somewhere - but I can't se it

Br

Gerth

main.cpp (3.06 KB)

[Robin2]

gerthe:
Enough background

It's not enough, or at least it was not presented clearly.

Please provide an example of two actual messages. As far as I can tell they are something like this

78 78 78 78 78 78 7a 01 c5 54 da 41  z 1 27.29139

but I don't know if the z 1 27.29139 is part of the message or where it comes from.

Also you have not said if there is an end-marker for every message - or perhaps a specific time interval between messages.

If you just need to watch out for 6 bytes with the value 78 and then capture the subsequent 6 bytes then it should be straightforward.

...R

[gerthe]

Hi Robin2 - thanks for your reply

This is a replay of info I got from the program CondorUDP2Com that I have captured with Termite.

78 78 78 78 78 78
7a 01 d5 54 da 41
7a 02 f6 66 79 44
7a 03 b0 55 54 bf
7a 04 d9 55 54 bf
78 78 78 78 78 78
7a 01 d1 54 da 41
7a 02 f6 66 79 44
7a 03 86 55 54 bf
7a 04 bc 55 54 bf
78 78 78 78 78 78
7a 01 cb 54 da 41
7a 02 f6 66 79 44
7a 03 b0 55 54 bf
7a 04 d6 55 54 bf

• The first row is the beginning of a data frame indicated by 6 bytes with 78 (x in HEX)
• Second row is the first "data row" it starts with a 7a (z in HEX) which just a marker and can be discarded, followed by the identifier for the parameter 01 (INT) , the upcoming 4 bytes are the data I require (Float, Little Endian)
• Subsetquent rows are data for parameter 02, 03 and 04 (can be up to 40 parameters in total)
• Row 6 - a new header for a new data frame, and so it repeats

So from the raw data above am I interested in the identifier i.e 01,02,03 ..... to know which step motor to move and the float value in the 4 upcoming bytes that is the momentary value for the parameter.

So this row from my first example

7a 01 c5 54 da 41 z 1 27.29139 // Speed in knots

Should have been interpreted as
7a 01 c5 54 da 41 which in plain text is z 1 27.29139, and parameter 01 = Speed in knots
Sorry for the confusion - it's so clear for myself after staring at it for hours.

also you have not said if there is an end-marker for every message - or perhaps a specific time interval between messages.

I can manipulate the frequency of the UPD stream coming to the program CondorUDP2Com , for the moment is that program updated each 50 ms.

If i look into the code from CondorUDP2Com that builds the UDP stream does it look like there is a "\n" after each block of 6 bytes. But that program is written in Python, another shortcoming in my programming knowledge.
Have attached the Python code , and according to the developer is the code block "formatFrame" the interesting part.

If you just need to watch out for 6 bytes with the value 78 and then capture the subsequent 6 bytes then it should be straightforward.

That was wat I also thought when I headed into this project but for the moment isn't I'm even able to fetch the first bye correct :o

Br

Gerth

CondorUDP2COM.txt (11.2 KB)

[Robin2]

In this

This is a replay of info I got from the program CondorUDP2Com that I have captured with Termite.

78 78 78 78 78 78
7a 01 d5 54 da 41
7a 02 f6 66 79 44
7a 03 b0 55 54 bf
7a 04 d9 55 54 bf

why is each 6 byte chunk on a different line? It suggests there is carriage-return and line-feed between them but I think that is unlikely. I had expected to see the data like this

78 78 78 78 78 78 7a 01 d5 54 da 41 7a 02 f6 66 79 44 7a 03 b0 55 54 bf 7a 04 d9 55 54 bf 78 78 78 78 78 78 .....

When I wrote Reply #1 I incorrectly thought that there was only a single 6-byte chunk after the 78 78 78 line but the same idea applies - watch out for the 6 78s and then save the next 24 bytes.

The simplest way to parse the 4 chunks of data is probably to save them to an array of structs. The content of the struct might be

struct MyDataStruct {
   byte firstByte;
   byte motorID;
   float motorMove;
};

But that assumes that the float is set in the format the Arduino uses. If not you will need to write your own code to interpret the 4 bytes that make up the float.

...R

[gerthe]

Hi Robin2

If the data stream is a continuous flow of bytes or is it's in chunks are completely irrelevant to me for the moment.
And I have a quite good idea about how to handle the data when that time comes as the developer of CondorUDP2Com have given an example program example on how to decode the stream.
Though is that example from an version where the start 78 78 78 78 78 78 chunk isn't sent - thus why I need to do my own code.

The core question - and that's what I asked for in my first post is why my Arduino don't interpret the data correct.

To broaden the view - The application CondorUDP2Com have a possibility to show both the raw data (hex) and the interpreted values, this Youtube video shows the application that's in between the simulator and my Arduino in action.

As seen in the right part of the applications screen, on the top is the COM stream that's sent out & at the bottom left is the values decoded , i.e what I suppose that I should be able to get from my Arduino.

The major problem I have is that my Arduino Mega 2650 don't decode correct.

In my code do I use
Serial.begin(9600); // Initialize USB-com for sending debug info
Serial1.begin(19200); //Initialization of incoming data via serial from CondorUDP2Com

I use Serial1.read() to fetch the first byte - in my experiments do I use a serial port sender to have control over what I send, but the results are the same as if i listen into the data stream directly, but when I print it out via Serial.print() to the debug does that value I get not correspond to the expected.

If i send in 0x78 so do I expect to ether have the char 'x' or the decimal value of 120 according to this ascii table

But my Aduino gives me decimal 232.

If I cant even decode the first byte correctly is further development not possible .

The serial terminal Termite do decode the data from CondorUDP2Com into

78 78 78 78 78 78 xxxxxx
7a 01 f9 54 da 41 z.ùTÚA
7a 02 f6 66 79 44 z.öfyD
7a 03 c3 55 54 bf z.ÃUT¿
7a 04 e3 55 54 bf z.ãUT¿
78 78 78 78 78 78 xxxxxx
7a 01 f3 54 da 41 z.óTÚA

From this do I draw the conclusion that a 78 = x (DEC 120) and a 7a = z and so on and this correspond to the ASCII table values

But when I send 0x78 to my application does it print out to my Serial.print() as decimal 232.
It's a big difference between assuming to fetch/test for an 'x' or DEC 120 but you receive DEC 232 (HEX E8, 'è')

To fault find this even more have I been experimenting with the bare bone example of Serial.read().
The outcome of this experiments are not conclusive either as they don't gives the same answers as my own code - but on the other hand it don't give me what I expect either.

According to the Arduino documentation on Serial.read() so is the return from that an integer thus do I assume that if 0x78 is read by Serial.read() so should the outcome of it be 120 , but in this experiment do I get 8 respectively 67 when I send 0x78 & 0x79.
And this boils back to the core question - Why don't I get 120 and 121 in return?

Please look on the attached image - this is the bare bone example of Serial.read() from the Arduino documentation.
Here do I use Termite (upper right) to send 2 bytes , Serial window on the bottom right is output

If I run the example code and send and receive via the serial port only (not serial1) do I get a 120 response if I send in a single 'x' , but if I send in 0x78 so do the output end up as 4 integers in the response, but i assume that 0x78 isn't a single byte when sent via the Serial monitor but 4 bytes.

[sterretje]

Maybe it's a good idea to post your Arduino code.

PS
Screenshots of code make it difficult to copy

[Robin2]

gerthe:
If the data stream is a continuous flow of bytes or is it's in chunks are completely irrelevant to me for the moment.

But maybe its not irrelevant for your Arduino program. And it's definitely not irrelevant for the understanding of people trying to help you.

Please post your program.

...R

[gerthe]

Hi sterretje & Robin2

I appreciate your attempts to help me out, don't feel anything else.

Attached is 2 pieces of code.

UDP_parser.ino is the code in the screenshot above and main.ino is the code I had before I started to loose my mind about how Serial1.read() should work. The two programs are very similar.

UPD_parser is nothing else than an test I did to make it very bare bone and read data from a Serial1 port and write out the data to the serial monitor via the Serial (usb) connector to try to figure out what's happening.

UDP_Parser.ino (573 Bytes)

main.ino (3.09 KB)

[sterretje]

Please note how to post code using code tags (limit is 9000 chars for a post). Can't look now.

UDP_Parser.ino

int incomingByte = 0; // for incoming serial data

void setup() {
  Serial.begin(9600);        // opens serial port, sets data rate to 9600 bps
  Serial1.begin(19200);
  delay(5);                  // stabilize
  Serial.println("Ready to roll");
}

void loop() {
  // send data only when you receive data:
  if (Serial1.available() > 0) {
    Serial.println("Got something");
    // read the incoming byte:
    incomingByte = Serial1.read();

    // say what you got:
    Serial.print("I received: ");
    Serial.println(incomingByte, DEC);
    
  }
}

main.ino

/*******************************************************
 * Author: Gerth Ericsson
 * Version:
 * Release date:
 * 
 * Program to fetch data from the application CondorUDP2COM to 
 * fetch data from Condor soaring 2 and drive stepper motors in 
 * an simulator.
 * 
 *******************************************************/ 

/*******************************************************
 * Data is sent from CondorUDP2Com at a rate of one dataframe each 50 ms
 * data rate is set in the file UDP.ini for condor.
 * 
 * Structure of a data block
 * HEX					real data		Note
 * -----------------	-------------	------------------	
 * 78 78 78 78 78 78  	xxxxxx
 * 7a 01 c5 54 da 41  	z 1 27.29139	// Speed in knots
 * 7a 02 f6 66 79 44  	z 2 997.60876 	// Altitude in feet
 * 7a 03 e2 55 54 bf  	z 3 -0.829443	// Vario m/s ??
 * 7a 04 f4 55 54 bf  	z 4 -0.829443	// Electric vario m/s ??
 * 
 * z 		= Start of data (HEX).
 * 1 - 40 	= Parameter index (int).
 * Data 	= Value for the index (float, little Endian).
 *****************************************************/

#include <Arduino.h>
#include <ArduinoTrace.h>

// Setup of variables
	byte data[40] 			= {0};		// Array to hold the 40 possible variables to get from CondorUDP2com.
	uint16_t incommingByte[2] 	= {};		// Array to hold all bytes to be parsed.
	byte rc					= 0;		// read byte from the serial port;


// Booleans
	bool inHeader 	= false;		// Marker if we are in the header.
	bool inData		= false;		// Marker if we are in the data part. 
	bool sync		= true;			// Start syncronization phase

// Integers
	int Param	 	= 0;			// Keep the current parameter.
	int idx         = 1;			// Counter index
  	int v3_hex 		= 0x78;			// test value to see what the heck happens
// Characters


void delayTimer(int ms) {
	// Whait 20 milliseconds to let the serial buffer to fill
	delay(ms);
	sync = false;

	return;
}

void setup() {

/* Set up serial ports to use
* incomming data is on Serial port 1 , tx 18, rx 19
* Serial used to monitor output
*/

  Serial.begin(9600);       			// Debug window
  Serial1.begin (19200);    // Input from CondorUDP2Com
  delay(5);								// Let intialization stabilise
  
  Serial.println("Setup finished");



}

void loop() {

 if (sync == true) {
	 delayTimer(20);
 }

 if (Serial1.available() > 0) {
	 // As the information will arrive in byte by byte so ?????.
	 // First will we need to find and discard the header for each data frame, 
	 // A frame header is composed of 6 byte 0x78
	
	Serial.print("Available bytes: ");
	Serial.println(Serial1.available());

	rc = Serial1.read();			// Lets read the first byte.
	//rc = v3_hex;
	Serial.print("Read hex: ");
	Serial.println(rc, HEX);
	Serial.print("Read DEC: ");
	Serial.println(rc, DEC);

	// if(rc != 120 ) {
	// 	// We have a header , discard block
	// 	Serial.print("We have a header char, discard");
	// } else {
	// 	Serial.print("We have found a data block , parse it.");
	// 	// Do some stuff when it works......
	// }
	 
	Serial.println((String) idx + " loop(s) done");
	Serial.println();
	 idx++;
	 
	}
}

[sterretje]

What is this CondorUDP2Com? Where did you get it from? Is there a spec?

[Robin2]

The program called UDP_Parser.ino is badly misnamed. All it does is read a byte from Serial1 and display the byte on Serial. It has nothing to do with UDP and it does not parse anything.

I don't know what was the intended purpose of the program called main.ino. It certainly makes no attempt to capture data of the format described earlier in this Thread.

Like I said in Reply #3 you need a program that looks out for the 6 x 0x78 bytes in a row and when it gets them it needs to save the subsequent 24 bytes .

Try something like this

byte body[24];
byte bodyCount = 0;
byte headerCount = 0;
if (Serial1.available() {
    inByte = Serial1.read();
    if (headerCount < 6) {
        if (inByte == 0x78) {
            headerCount++;
            if (headerCount == 6) {
                bodyCount = 0;
            }
        }
        else {
            headerCount = 0;
        }
    }
    else {
        if (bodyCount < 24) {
            body[bodyCount] = inByte;
            bodyCount ++;
        }
        else {
            for (byte n = 0; n < bodyCount; n++) {
                Serial.println(body[n]);
            }
            headerCount = 0; // ready for next message
        }
    }
}

...R

[cattledog]

If I run the example code and send and receive via the serial port only (not serial1) do I get a 120 response if I send in a single 'x' , but if I send in 0x78 so do the output end up as 4 integers in the response, but i assume that 0x78 isn't a single byte when sent via the Serial monitor but 4 bytes.

You can not send hex from the Serial monitor. You have to use a terminal program like you are doing. I like using Cool Term.

What is troubling me is that when you do send hex values from Termite in an effort to emulate the Condor you can not read the values in the simple test code.

Are you certain about the Termite to Serial 1 Arduino link?

[gerthe]

Hi Cattledog

What is troubling me is that when you do send hex values from Termite in an effort to emulate the Condor you can not read the values in the simple test code.

You have got my dilemma correct - my code is very barebone due to the fact that I have not been able to fetch the simplest HEX code inbound, thus have I redused it just this code.

If i'm not able to fetch a simple 0x78 at this stage is there no need to try to evolve the code into the other tasks it need to do.

I Will download Cool Term and see if it gives me some other result.

[cattledog]

You are probably going to need a usb to ttl serial converter between the terminal and the Mega Serial 1.

[gerthe]

I have downloaded and tested Cool term as well as testing Serial1, Serial2 and Serial3 & two different USB to Serial adapters - And no luck the result is the same as earlier.

The only thing I haven't made any change to is the home made Serial cable I have.

I have taken a Serial cable and soldered pins to the cables for RX, TX and GND.

[cattledog]

Perhaps the cable you modified is a cross over cable or if that's what you were expecting maybe its not a cross over cable.

What happens if you switch the RX/TX connections?

[gerthe]

I'm quite confident that there is no problem with the cable , I have measured the Rx/Tx pin to the correct sockets in the DSub

If I switch the TX/RX around don't the Arduino react at all.

After your hint about a TTL converter have I found a couple of videos how to use another Arduino as a USB>TTL converter.
I have a small Arduino Uno at hand and I will try to follow the instructions and turn that one into a converter and see if that rectifies my odd problem.
if that works I will need to go out and buy some more robust device to use in the project.

But that will be a task for tomorrow.

[gerthe]

Hi Sterretje

A small descripton of the application can be found here - CondorUDP2Com

Also is the source code on GitHub via this link

Br

Gerth

[sterretje]

What is the silver cable / blue connector in the front? A USB-to-RS232 converter? Connected to your PC?

What is the black cable / black connector in the front. As it is a subD, it probably carries RS232 signals and not TTL signals if there are no active components in there. Is that cable connected to your board?

If the above is true, it will not work; you will need a RS232-to-TTL converter.

[gerthe]

Hi sterretje

Yes you are correct.

The blue device with the silver cable is an USB to Serial converter and I use it to connect to the computer where i have the serial sender or CondorUDP2Com application.

The black connector is one of the ends on the cable I use, the other end is directly connected to the Arduino board via the blue and white cable in connector in socket 17 & 16 (in this case Serial2 on the Mega) , do also use GND in the cable.
But it's straight from the serial converter to the board without anything in between.