Plucking portions of data from serial stream to display on LCD

Hello all,

I am new to coding and Arduino and am trying my best to get up to speed. I just started delving into this about three weeks ago.

My project involves interfacing to a two way radio to emulate a control head. So far I have the buttons and various other control working fine. My next hurdle is to properly display the information which would be displayed on the control head. The protocol is regular RS232 but it's in hex strings which are formatted in a certain way. My thought was to circumvent having to do full packet parsing and parsing and just flag for the first few hex bits. The first few characters of each string will be similar but will be specific to each of the four or five strings and only differ by one bit which is the last one 0x03 in the code I posted. A different line would be the same information but with 0x04 etc.... the variable checksum is toward the very end of the packets and it has an end of line marker but I was thinking that if I flag for those first few and I know the length then I shouldn't have to worry about checksums and watching for end of lilnes right?

I'm pretty sure I need to use a substring to do this but I'm getting lost in how to set that up properly and how to position the data on the display in the place I want. It seems to want to just print to one character place when I try to utilize lcd.setCursor(col, row)

This code is probably very ugly but seems to be working as intended to at least sniff out strings that match the parameters set.

(firstchar = Serial.read()); {
if (firstchar > 0) { //get incoming byte
if (firstchar == 0x24,0x0F,0x81,0x2E,0x03) { // data beyon 0x03 is hex encoded display information such as frequency number 10,000.00
byte acknowledge[6] = {0x24,0x01,0x10,0xF3,0x28,0x03}; // Ack packet to initiate next string of data from the device when information is received
Serial.write(acknowledge, 6); // will write the array of bytes to the serial port in raw hex form length and keep the data flowing as long as the display information is updated (changes from radio)
char buffer[34];
delay(22);
for (i=0;i<34;i++) {
buffer = Serial.read();

• String frequencyString = buffer;*
  _ lcd.write(buffer*);*_
  The packet continues on past the 0x03 and includes ten characters of hex encoded display information. I would like to somehow grab the data from spots 7 through 17 (counting space 0 of course) and display only that information on the LCD. As it is now it is just flooding the LCD with the information and writing the display frequency over and over starting with LCD row 0 then goes to row 2 then jumps back up to row 1 and ends at row 3 with a lot of garbage intermixed.
  I hope this makes sense.
  Thanks in advance.

First off it sounds like you are trying to fix several problems at the same time.

First Off - worry only about the serial data you are trying to intercept.
Are you getting data for more than one line on the display?

look at the setCursor() command - Before you put anything to the display make sure you put the cursor at the proper position. You may also have to print some strings of spaces to clear old data (you print 6 chars where 8 were before...

Don't use println. That causes the cursor to move to a different line. Use setCursor and then print.

Does the data have some character that is always in the same position in the serial data (carriage return at the end is always nice) In Setup() have a routine that reads the serial data until that character appears and then go to loop().

The data breaks down as follows, example:

24 0F 81 2E 03 02 6E 46 20 20 38 2C 39 39 31 2E 39 38 81 03
$.... nF 8,991.98.

All lines start with 0x24
0x0F indicates length of data packet (less checksum 0x81 and end of line marker bit 0x03)
0x81 denotes packet is outgoing traffic
0x2E opcode for display
0x03 subopcode for certain line of display (second line down)
0x02 misc data that is not pertinent
0x6E more non pertinent data
0x46-0x38 is the actual display info (F 8,991.98)
0x81 checksum (all data summed less the checksum itself of course and the 0x03 end of line marker)
0x03 end of line marker

There are a few other similar data packets of interest that are structured similarly but of different lengths and carry soft button labels and other display information which I would want to display as well. I could of course just use whatever method works for this example for any other lines.

My thought was to forego the actual checksum and verification process and instead look for the above line by watching for the string 0x24 0x0F 0x81 0x2E 0x03 and flagging that as a start of line indication since other display lines would differ by the sub op code (0x03 in above example) and would thus be unique as well. The next step would be to load that flagged string into a buffer set to the maximum length any of the information strings will every comprise. Then count over in the data stored to the actual display portion which comprises the (F 9,991.98), display that on lcd row 1, column 0 and discard the rest.

I hope this makes more sense. I am seeing the above frequency information on the LCD as it is now each time I tune the unit but it's flooding the LCD and writing it all over the display in a staggered fashion with the other data mixed in of course since nothing is being trimmed down and formatted in any way.

I believe I should probably look for the above mentioned strings, save them off in an array or buffer then extract the portions of interest, then display that to the LCD then clear any buffers but I'm not to a point yet that I can sit down and just write that. I am not trying to be greedy by looking for someone to do it all for me, just a few pointers to get me steered toward the appropriate logic to use. Or, perhaps I should go a totally different direction than what I'm proposing?

kf2qd:
First off it sounds like you are trying to fix several problems at the same time.

First Off - worry only about the serial data you are trying to intercept. Agreed this is essential
Are you getting data for more than one line on the display? Yes, upon initial turn on it does query for all display content and I see it go by. Obviously this means my method so far must not be working as an appropriate "filter".

look at the setCursor() command - Before you put anything to the display make sure you put the cursor at the proper position. You may also have to print some strings of spaces to clear old data (you print 6 chars where 8 were before... OK, I was thinking in terms of some sort of buffer flush but I see your thought process.

Don't use println. That causes the cursor to move to a different line. Use setCursor and then print. I had avoided printlin for that reason but it seemed when I used just print instead of write I ended up with the data being serially streamed through only one LCD characters space sequentially instead of printed out laterally.

Does the data have some character that is always in the same position in the serial data (carriage return at the end is always nice) In Setup() have a routine that reads the serial data until that character appears and then go to loop().I kind of answered your posts backward. But to answer it directy, yes, 0x03 is end of line. However, as you will see in my breakdown of the packets 0x03 could be used numerous tiimes for other activity so simply flagging for that is not an option unless the line length bit is accounted for and factored and as you can see there are no delimeters. It's all straight hex.

I'm having difficulty figuring out how to display only the portions of the data I want to see in the string that is read out.

if (firstchar = Serial.read()) {
if (firstchar > 0) { //get incoming byte
if (firstchar == 0x24,0x0F,0x81,0x2E,0x03) { //unique header of data packet
byte acknowledge[6] = {0x24,0x01,0x10,0xF3,0x28,0x03}; // Ack packet to initiate next string of data otherwise device will stall and not report any more data when a button is pushed again
Serial.write(acknowledge, 6); // will write the array of bytes to the serial port that compose the acknowledge packet
char buffer[20]; //buffer to hold the incoming data packet that comprises the line with the frequency readout
delay(22); //delay to allow data to get in buffer
for (i=0;i<20;i++) { //for statement to set buffer constraints
buffer = Serial.read(); *
_ lcd.print(buffer); //prints the buffer data to the LCD, I am seeing stuff like $blocks of asian like text.more asian textR 11,173.00->more asian text~ etc.... overwriting previous data on LCD*_
* //where similar data had been printed but with previous frequency numbers such as R 11,172.00-> etc....*
}[/size]
So, I would like to strip out that R 11,175.00 and place that on LCD line 1 column 0.
I have been reading and studying examples of similar projects but I am just not understanding how to place the received data into something so I can somehow strip out the useful parts and discard the rest.
Thanks in advance for any help.

I'm having difficulty figuring out how to display only the portions of the data I want to see in the string that is read out.

First, that code you posted is a mess. Comments do not have to be placed at the end of a line of code. They CAN be placed on a line before the code. If the comment merely states the obvious, get rid of it.

         if (firstchar == 0x24,0x0F,0x81,0x2E,0x03) {            //unique header of data packet

Garbage. The , operator is not doing what you seem to think it does, although it is very hard to tell what you think this is doing. Whatever it is you think this is doing, it isn't.

The Tools + Auto Format menu item needs to become your friend, as does the # icon for posting code.

            for (i=0;i<20;i++) {     //for statement to set buffer constraints
               buffer[i] = Serial.read();

More garbage. The comment is nonsense. You send a request for acknowledgement, according to the comment, and expect an instantaneous response of two characters to all have arrived nanoseconds later. Serial data doesn't work that way.

//prints the buffer data to the LCD, I am seeing stuff like $blocks of asian like text.more asian textR

Another statement of the obvious. Of course you are seeing garbage on the screen, for several reasons. One is that the Serial.read() function returns -1 if there is nothing to read. You store that in the array, and expect to print it on the LCD as though it was a printable character. It isn't.

Second is that the lcd.print() function expects a string. You are passing it an array of characters, which is NOT the same thing. A string is an array of characters THAT IS NULL TERMINATED. Your array of characters is not NULL terminated, therefore it is not a string. Therefore you should not be passing it to a function that expects a string.

There are bazillions of examples around for reading and buffering serial data as strings. Time to do some searching.

Don't even bother with trying to make use of the data until you know how to wait for, read, and store the data properly.

The below might be of interest. is the data sent as individual bytes or is each byte represented by a four character hex representation of the byte?

http://www.arduino.cc/playground/Code/TextFinder

PaulS:

..........
There are bazillions of examples around for reading and buffering serial data as strings. Time to do some searching.

Don't even bother with trying to make use of the data until you know how to wait for, read, and store the data properly.

Yea, that pretty much sums it up.

I have the output part working pretty well (buttons and digital volume) but the concepts of strings and arrays quite obviously has me lost.

So to start off small how would you suggest I look for the initial few hex characters that are unique to the start of the data packet of interest? I see examples of one single character being flagged as a start of line but haven't found a-lot on the idea of looking for more.

Thanks for the reply. I am glad that I need to throw everything out and just start over and learn much more about the basics.

Thanks, I will check that out.

Maybe seeing what I am seeing will make more sense. If I sniff the serial traffic this is what I see back for the frequency display line coming in as individual bytes.

24 0F 81 2E 03 02 6E 46 20 20 38 2C 39 39 31 2E 39 38 81 03
$.... nF 8,991.98.

zoomkat:
The below might be of interest. is the data sent as individual bytes or is each byte represented by a four character hex representation of the byte?

Arduino Playground - TextFinder

If you know that you want to read 20 characters, you need to either read them as they become available, or wait for them to become available, and then read them all at once. The second is simplest.

     Serial.write(acknowledge, 6);
     char buffer[21];

     while(Serial.available() < 20)
     {
        // Do nothing
     }

     for (i=0;i<20;i++)
     {
         buffer[i] = Serial.read();
         buffer[i+1] = '\0';
     }     
     lcd.print(buffer);

(Useless and incorrect comments deleted)

I envy your knowledge. It would be nice not to feel like like I have a mental blindfold on while thrashing around like an idiot. There just seems to be so much to learn and so many concepts it's overwhelming.

Keep at it for 32 years. I doubt you'll still feel like an idiot.

I don't know. Some days...

I've had a few of those "what the hell was I thinking?" days myself. Not nearly as stressful as first learning a language, though. At least, I can now recognize a mistake when I make it.

Ok, with that I am able to see the information but of course, it is strewn across the LCD and not placed in any certain position. Also, the LCD has all pixels lit. The information flashes over the lit areas and then it returns to an all on state.

So, with that said any pointers on which direction to head when it comes to actually pulling out the ten character fields F 8,992.00 in my above example?

Thanks again.

mancow:
If you know that you want to read 20 characters, you need to either read them as they become available, or wait for them to become available, and then read them all at once. The second is simplest.

The characters look like all bits on because you don't have the contrast set properly.

No, I thought that too but that's not it. The activity is the same regardless of the setting of the trim pot on the serial backpack board, only it's in proportion to whatever the setting is.

**edit

It's just overrunning the LCD. I stripped things down to the bare basics, took out the volume control code etc... and am focusing on the display activity. I am sending test codes to it from a serial program and it's printing the text fine now because the test data is matching the constraints set in the code.

Now, I need to figure out how to strip off the unwanted data and only display the portions I want to see from the radio.

I'm probably just babbling and spamming the board here so I should probably just keep quiet until I figure something out useful I suppose.

Now, I need to figure out how to strip off the unwanted data and only display the portions I want to see from the radio.

And your current code looks like?

I had the forum set to the quick reply and didn't see the # function earlier.

As for the code, well I really don't have anything other to work with than what you posted earlier.

     char buffer[21];

     while(Serial.available() < 20)
     {
        // Do nothing
     }

     for (i=0;i<20;i++)
     {
         buffer[i] = Serial.read();
         buffer[i+1] = '\0';
     }     
     lcd.print(buffer);

I saved off all my other working control code and have been starting with a very basic blank sketch based on this except I'm using Serial.print instead of lcd so I don't have to mess with LCD stuff right now. I've been trying to read as much as possible and have like 15 tabs of different references and various web pages up but I'm still trying to get the basic concepts.

Using the code above it will of course echo back to the serial the 20 hex characters I send to it. I am trying to figure out the next small step of how to work with the data it sends back. I am trying to figure out how to send back only portions of what is received such as bytes 5 through 10 or something similar.

Using the code above it will of course echo back to the serial the 20 hex characters I send to it. I am trying to figure out the next small step of how to work with the data it sends back. I am trying to figure out how to send back only portions of what is received such as bytes 5 through 10 or something similar.

Arrays are arrays. Doesn't matter if they contain chars, ints, floats, pointers, or structs. If you want elements 5 to 10 from one array in another one, create the 2nd array, and copy the data.

char importantStuff[10];
for(int i=5, j=0; i<=10; i++, j++)
{
   importantStuff[j] = buffer[i]; // Copy a character
   importantStuff[j+1] = '\0'; // Keep NULL terminated
}
Serial.print("importantStuff: [");
Serial.print(importantStuff);
Serial.println("]");

If you are not sure where the data of interest is (5 to 10 may not be quite right), you can use strchr() to find a character in a string (a comma, maybe) or strstr() to find a string in a string ("LED=", maybe).