Pages: [1] 2   Go Down
Author Topic: Parsing serial data sent to an arduino  (Read 16522 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,

I am trying to take a comma separated (and null terminated i think) string sent to an arduino via the USB serial interface and then output it via an attached LCD. The LCD part works fine but i can't figure out how i would parse a comma delimited string and extract the individual values on  an arduino. The strings are coming from an application that is taking status information from flight simulator (engine rpm, altitude, etc) and is outputted by the application in the following format:

Code:
2618,316,109,00,2527,4

where:

- 2618 is the altitude
- 316 is the heading
- 109 is the airspeed
- 00 is the flap position in degrees
- 2527 is the engine rpm
- 4 is a number that just increases by 1 from 0 - 9 every second .... not sure why, but it does. No option to turn it off either.

The output from the program as observed from putty is as follows:

Code:
2618,316,109,00,2527,4
2618,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2620,316,109,00,2527,4
2620,316,109,00,2527,4
2620,316,109,00,2527,5
2620,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2622,316,109,00,2528,5
2622,316,109,00,2528,5
2622,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2624,316,109,00,2528,5
2624,316,109,00,2528,5
2624,316,109,00,2528,5
2625,316,109,00,2528,5
2625,316,109,00,2528,5
2625,316,109,00,2528,5
2626,316,109,00,2528,5
2626,316,109,00,2528,5
2626,316,109,00,2528,5
2627,316,109,00,2528,5
2627,316,109,00,2528,5
2627,316,109,00,2528,5
2628,316,109,00,2528,5
2628,316,109,00,2528,5
2629,316,109,00,2528,5
2629,316,109,00,2527,5
2629,316,109,00,2527,5
2630,316,108,00,2527,5
2630,316,108,00,2527,5
2631,316,108,00,2527,5
2631,316,108,00,2527,5
2631,316,108,00,2527,5
2632,316,108,00,2527,5
2632,316,108,00,2527,5
2633,316,108,00,2527,5
2633,316,108,00,2527,6
2633,316,108,00,2527,6
2634,316,108,00,2527,6
2634,316,108,00,2527,6
2635,316,108,00,2527,6
2635,316,108,00,2527,6
2636,316,108,00,2526,6
2636,316,108,00,2526,6
2637,316,108,00,2526,6
2637,316,108,00,2526,6
2638,316,108,00,2526,6
2638,316,108,00,2526,6
2638,316,108,00,2526,6
2639,316,108,00,2526,6
2639,316,108,00,2526,6
2640,316,108,00,2525,6

and so on if you get the idea. My question is now this:

- how do i read a comma delimited string via serial and parse it into separate variables. Also would the strings in this case be null terminated ?
- will an arduino be able to handle this data? (it is being sent at 115200 which i know it supports but can it take it via the usb interface from the computer as a constant stream of data?)
-  how i do i read a continuous stream of serial data and parse them as i go along. (even if i have to skip or ignore a few lines i just need it to wait for the beginning of the next available line, parse it, and extract the values)

I think this should be an easier method as it is just comma separated numbers but then again i am new to arduino serial communication so i would appreciate any help with this. Even if it's only how to parse and extract the values. The LCD part i have no problems with.
Logged

0
Offline Offline
Newbie
*
Karma: 1
Posts: 12
Arduino & Raspberry Pi trainer, hardware hacker, software slave...
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Am just testing this myself so no promises but have a look at the Messenger library which looks like it should do the trick.

Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
how do i read a comma delimited string via serial
Just like any other string. There are plenty of examples around for reading a string from the serial port.

Quote
parse it into separate variables.
The strtok function is just what you need.

Quote
Also would the strings in this case be null terminated ?
Which strings? The ones returned by strtok? Those will be NULL terminated. Or do you mean the one read from the serial port. That string will be NULL terminated only if you make it so.

Quote
will an arduino be able to handle this data?
If by "handle it" you mean will the Arduino be able to read data from the serial port, the answer is yes. If by "handle it" you mean rough it up and mistreat it, that's possible, too.

Quote
how i do i read a continuous stream of serial data and parse them as i go along.
In psuedo code:
Is there serial data?
If so, read the next character.
Is it a comma?
If so, convert the string already read to a number, set the index to 0, and put a NULL in the location pointed to be index. Do something with the number.
Otherwise, store the character in the array, at the location pointed to be index, increment index, and store a NULL at the location pointed to by index.

On the other hand, there is a serial buffer, so it is not necessary to grab every character as it arrives. If you can "skip a few lines" this implies that there is some end-of-record marker that denotes a line.

Read a whole line from the serial port, stopping when you have read that end-of-record marker. Process the string read in, and repeat.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

First off thanks for all the replies,

Hi again and thanks PaulS. Yeah I'm not really sure how the strings are terminated in that output nor am i sure how to find out. Would be nice as then i could say "read until" or "start reading after the next" in that case i could have the arduino start reading from the serial stream at a certain point, stop at a certain point, deal with the data and then start reading from the next line start. Unfortunately i have no control of the serial output so i was wondering if line starts and ends could be differentiated using the data in it's current state.

As for strtok i found an example that can extract data from a string using that method earlier but alas i would need to have the string extracted from the stream. The code as the example had it was:

Code:
#include <string.h>

char sz[] = "3500,270,890,70,4";
void setup()
{
 char *p = sz;
 char *str;
 Serial.begin(115200);
 while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the semicolon
   Serial.println(str);
}

void loop(){}

(can't recall where i got that, so if anyone could send me a source that'll be nice)

my idea for that piece of code would be to somehow get a single line of data out from the serial output in a loop and for each run of the loop do the code above. Hence also in psuedo code it would do the following:
- If serial is available
-- Read the serial data up to or from a point (not sure if i can actually differentiate the start and end points of the data yet) and add it to a string.
-- Take the resulting string and send it to the strtok code snippet posted above
-- Have that loop through the string and seperate it into values (preferably numbers but doesn't really matter) and add those values to individual variables (probably var1, var2 as incremented by the internal loop counter)
-- Take the resulting variables and display them
- Rinse and repeat

I still need to get a handle on the strtok function but i believe i understand it enough to work with it. However my main (and seemingly re-occurring issue ) is how to get individual string lines from the serial input stream.

  
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The putty output makes it appear as though the serial data stream contains carriage returns and/or line feeds. Those can be used as end-of-record markers.

This code will read the data into an array, up to the end-of-record marker
Code:
char inData[80];
byte index = 0;

void loop()
{
   while(Serial.available() > 0)
   {
      char aChar = Serial.read();
      if(aChar == '\n')
      {
         // End of record detected. Time to parse

         index = 0;
         inData[index] = NULL;
      }
      else
      {
         inData[index] = aChar;
         index++;
         inData[inex] = '\0'; // Keep the string NULL terminated
      }
   }
}

At the appropriate spot (see the comment), put the code to parse the string.

The string parsing example you posted has some issues. The comment says that the delimiter is a semicolon, while in reality it is a comma.

That code will also return the first value over and over. To make it return all the values, p needs to be set to NULL after the first call to strtok:
Code:
while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   Serial.println(str);
   p = NULL;
}
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Wow Thanks alot paul,

About the semicolon and comma thing, i modified it to a comma in my testing. Also in regards to the last code nulling "p" can i do something like this:

Code:
counter = 0; //initialise the counter
while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   storage[counter] = str; //use the counter as an index to add each value to the array
   counter++; //increment the counter
}
p = NULL; //clear the serial data storage outside of the loop after all the data has been retrieved
//using the liquid crystal library print out info
//kind of coding from memory so imagine that the cursor has been setup and everything
lcd.print("Speed: "); lcd.print(storage[0]); //print out the data stored in the value storage array index '0'
//set cursor to next line using appropiate commands again
lcd.print("Altitude: "); lcd.print(storage[1]); //ditto
// and so on and so on
//only thing i can figure now is to null the array by looping through and clearing it using a loop that repeats the number of times of "counter"

And that would be inserted in the "time to parse" section of the previous code. Does that seem like it could work reasonably? I am at work so i don't have my arduino with me so i'll have to test when i get back home. for now i'm trying to get the logic in order. Once again i really appreciate your help. This has given me a huge boost in getting this project working.
« Last Edit: August 02, 2010, 08:04:50 am by juerujin » Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The strtok function takes a pointer to the string to parse, or a NULL to keep parsing the same string. You need to move the assignment of NULL to p INSIDE the curly braces, so that p is NULL for the next call.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Ah okay that explains it. Other than that however is the code more or less workable? i.e.

Code:
counter = 0; //initialise the counter
while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   storage[counter] = str; //use the counter as an index to add each value to the array
   counter++; //increment the counter
   p = NULL;
}

//using the liquid crystal library print out info
//kind of coding from memory so imagine that the cursor has been setup and everything
lcd.print("Speed: "); lcd.print(storage[0]); //print out the data stored in the value storage array index '0'
//set cursor to next line using appropiate commands again
lcd.print("Altitude: "); lcd.print(storage[1]); //ditto
// and so on and so on
//only thing i can figure now is to null the array by looping through and clearing it using a loop that repeats the number of times of "counter"

Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Other than that however is the code more or less workable?
Only one way to really know. Upload and see what happens.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Heh that's true. Well thanks again for the help .... i'll let you guys know how it turned out when i get back.
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well i'm probably gonna get home late tonight so i might as well post the experimental code.

NOTE THIS CODE IS UNTESTED. I am just posting this to get any feedback before i run it:

Code:
#include <LiquidCrystal.h>
#include <string.h>
char inData[80];
char flightdata[80];
byte index = 0;
LiquidCrystal lcd(12, 11, 2, 3, 4, 5);

void setup()
{
Serial.begin(9600);
lcd.begin(20, 2);
}

void loop()
{
   while(Serial.available() > 0)
   {
      char aChar = Serial.read();
      if(aChar == '\n')
      {
         // End of record detected. Time to parse
            char *p = inData; //assign the string to *p
            char *str;        //intialize str
            int counter = 0; //initialise the counter
            
            while ((str = strtok_r(p, ",", &p)) != "\0") // delimiter is the comma. NULL is the terminator
            {
                  flightdata[counter] = *str; //use the counter as an index to add each value to the array
                  counter++; //increment the counter

               p = NULL;
            }
            lcd.print("Speed: "); lcd.print(flightdata[0]); //print out the data stored in the value storage array index '0'
            //set cursor to next line using appropiate commands again
            lcd.print("Altitude: "); lcd.print(flightdata[1]); //ditto



         index = 0;
         inData[index] = NULL;
          
      }
      else
      {
         inData[index] = aChar;
         index++;
         inData[index] = '\0'; // Keep the string NULL terminated
      }
   }
}





Let me know if i overlooked anything ..... it compiles okay but that doesn't mean anything.
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
           char *p = inData; //assign the string to *p
            char *str;        //intialize str
            int counter = 0; //initialise the counter
            
            while ((str = strtok_r(p, ",", &p)) != "\0") // delimiter is the comma. NULL is the terminator
Some of the comments are wrong. You are not assigning the string to *p. You are making p point to the string.

You are not initializing str. You are declaring it.

These are trivial. There is no reason to have the != "\0" stuff in there. The str variable will point to a string or will point to NULL. Any non-NULL pointer will be true.
Code:
while (str = strtok_r(p, ",", &p))
{
}
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

ah good point,

I'll make the edits and i'll try to stop being lazy when i comment .... believe me i understand the value of accurate commenting .... especially when you write a 300 line piece of code, go away for two months and then try to remember what you did smiley-kitty;;
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 22
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Well i tried the script and it was a bust so i broke it down into smaller and smaller elements until all i had was this

Code:
#include <LiquidCrystal.h>

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);

void setup() {
  // set up the LCD's number of rows and columns:
  lcd.begin(20, 4);
}

void loop() {
  while(Serial.available() > 0)
   {
      
  lcd.setCursor(0, 0);
  lcd.print("LAT:XXXXXXXXXXXXXXXX");
  lcd.setCursor(0, 1);
  lcd.print("LON:XXXXXXXXXXXXXXXX");
  lcd.setCursor(0, 2);
  lcd.print("SPEED:");
  lcd.print(millis()/1000);
  lcd.setCursor(13, 2);
  lcd.print("HDG:");
  lcd.print(millis()/1000);
  lcd.setCursor(0, 3);
  lcd.print("ALT:XXXXXXYYXXXXXXXX");
  // print the number of seconds since reset:
  delay(1000);
  
   }
  
}

no matter what the screen just stays blank. I even tried adding in a delay just in case everything was happening too quickly. Any ideas what i should try now?
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If the screen remains blank with this sketch, there are two possible causes. The first is that there is nothing sending the Arduino serial data. The second is that LCD is not working.

I presume you have verified that the LCD works.

If the LCD works, but no serial data is being received, it is not an Arduino software problem. The Arduino can not make serial data appear. The issue then is either a hardware problem or a software problem with the application that is sending the data or a communication problem (the Arduino and the sender are not getting exclusive access to the serial port).
Logged

Pages: [1] 2   Go Up
Jump to: