Parsing serial data sent to an arduino

Hi,

I am trying to take a comma separated (and null terminated i think) string sent to an arduino via the USB serial interface and then output it via an attached LCD. The LCD part works fine but i can't figure out how i would parse a comma delimited string and extract the individual values on an arduino. The strings are coming from an application that is taking status information from flight simulator (engine rpm, altitude, etc) and is outputted by the application in the following format:

2618,316,109,00,2527,4

where:

• 2618 is the altitude
• 316 is the heading
• 109 is the airspeed
• 00 is the flap position in degrees
• 2527 is the engine rpm
• 4 is a number that just increases by 1 from 0 - 9 every second .... not sure why, but it does. No option to turn it off either.

The output from the program as observed from putty is as follows:

2618,316,109,00,2527,4
2618,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2619,316,109,00,2527,4
2620,316,109,00,2527,4
2620,316,109,00,2527,4
2620,316,109,00,2527,5
2620,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2621,316,109,00,2528,5
2622,316,109,00,2528,5
2622,316,109,00,2528,5
2622,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2623,316,109,00,2528,5
2624,316,109,00,2528,5
2624,316,109,00,2528,5
2624,316,109,00,2528,5
2625,316,109,00,2528,5
2625,316,109,00,2528,5
2625,316,109,00,2528,5
2626,316,109,00,2528,5
2626,316,109,00,2528,5
2626,316,109,00,2528,5
2627,316,109,00,2528,5
2627,316,109,00,2528,5
2627,316,109,00,2528,5
2628,316,109,00,2528,5
2628,316,109,00,2528,5
2629,316,109,00,2528,5
2629,316,109,00,2527,5
2629,316,109,00,2527,5
2630,316,108,00,2527,5
2630,316,108,00,2527,5
2631,316,108,00,2527,5
2631,316,108,00,2527,5
2631,316,108,00,2527,5
2632,316,108,00,2527,5
2632,316,108,00,2527,5
2633,316,108,00,2527,5
2633,316,108,00,2527,6
2633,316,108,00,2527,6
2634,316,108,00,2527,6
2634,316,108,00,2527,6
2635,316,108,00,2527,6
2635,316,108,00,2527,6
2636,316,108,00,2526,6
2636,316,108,00,2526,6
2637,316,108,00,2526,6
2637,316,108,00,2526,6
2638,316,108,00,2526,6
2638,316,108,00,2526,6
2638,316,108,00,2526,6
2639,316,108,00,2526,6
2639,316,108,00,2526,6
2640,316,108,00,2525,6

and so on if you get the idea. My question is now this:

• how do i read a comma delimited string via serial and parse it into separate variables. Also would the strings in this case be null terminated ?
• will an arduino be able to handle this data? (it is being sent at 115200 which i know it supports but can it take it via the usb interface from the computer as a constant stream of data?)
• how i do i read a continuous stream of serial data and parse them as i go along. (even if i have to skip or ignore a few lines i just need it to wait for the beginning of the next available line, parse it, and extract the values)

I think this should be an easier method as it is just comma separated numbers but then again i am new to arduino serial communication so i would appreciate any help with this. Even if it's only how to parse and extract the values. The LCD part i have no problems with.

Am just testing this myself so no promises but have a look at the Messenger library which looks like it should do the trick.

how do i read a comma delimited string via serial

Just like any other string. There are plenty of examples around for reading a string from the serial port.

parse it into separate variables.

The strtok function is just what you need.

Also would the strings in this case be null terminated ?

Which strings? The ones returned by strtok? Those will be NULL terminated. Or do you mean the one read from the serial port. That string will be NULL terminated only if you make it so.

will an arduino be able to handle this data?

If by "handle it" you mean will the Arduino be able to read data from the serial port, the answer is yes. If by "handle it" you mean rough it up and mistreat it, that's possible, too.

how i do i read a continuous stream of serial data and parse them as i go along.

In psuedo code:
Is there serial data?
If so, read the next character.
Is it a comma?
If so, convert the string already read to a number, set the index to 0, and put a NULL in the location pointed to be index. Do something with the number.
Otherwise, store the character in the array, at the location pointed to be index, increment index, and store a NULL at the location pointed to by index.

On the other hand, there is a serial buffer, so it is not necessary to grab every character as it arrives. If you can "skip a few lines" this implies that there is some end-of-record marker that denotes a line.

Read a whole line from the serial port, stopping when you have read that end-of-record marker. Process the string read in, and repeat.

First off thanks for all the replies,

Hi again and thanks PaulS. Yeah I'm not really sure how the strings are terminated in that output nor am i sure how to find out. Would be nice as then i could say "read until" or "start reading after the next" in that case i could have the arduino start reading from the serial stream at a certain point, stop at a certain point, deal with the data and then start reading from the next line start. Unfortunately i have no control of the serial output so i was wondering if line starts and ends could be differentiated using the data in it's current state.

As for strtok i found an example that can extract data from a string using that method earlier but alas i would need to have the string extracted from the stream. The code as the example had it was:

#include <string.h>

char sz[] = "3500,270,890,70,4";
void setup()
{
 char *p = sz;
 char *str;
 Serial.begin(115200);
 while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the semicolon
   Serial.println(str);
}

void loop(){}

(can't recall where i got that, so if anyone could send me a source that'll be nice)

my idea for that piece of code would be to somehow get a single line of data out from the serial output in a loop and for each run of the loop do the code above. Hence also in psuedo code it would do the following:

• If serial is available
  -- Read the serial data up to or from a point (not sure if i can actually differentiate the start and end points of the data yet) and add it to a string.
  -- Take the resulting string and send it to the strtok code snippet posted above
  -- Have that loop through the string and seperate it into values (preferably numbers but doesn't really matter) and add those values to individual variables (probably var1, var2 as incremented by the internal loop counter)
  -- Take the resulting variables and display them
• Rinse and repeat

I still need to get a handle on the strtok function but i believe i understand it enough to work with it. However my main (and seemingly re-occurring issue ) is how to get individual string lines from the serial input stream.

The putty output makes it appear as though the serial data stream contains carriage returns and/or line feeds. Those can be used as end-of-record markers.

This code will read the data into an array, up to the end-of-record marker

char inData[80];
byte index = 0;

void loop()
{
   while(Serial.available() > 0)
   {
      char aChar = Serial.read();
      if(aChar == '\n')
      {
         // End of record detected. Time to parse

         index = 0;
         inData[index] = NULL;
      }
      else
      {
         inData[index] = aChar;
         index++;
         inData[inex] = '\0'; // Keep the string NULL terminated
      }
   }
}

At the appropriate spot (see the comment), put the code to parse the string.

The string parsing example you posted has some issues. The comment says that the delimiter is a semicolon, while in reality it is a comma.

That code will also return the first value over and over. To make it return all the values, p needs to be set to NULL after the first call to strtok:

while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   Serial.println(str);
   p = NULL;
}

Wow Thanks alot paul,

About the semicolon and comma thing, i modified it to a comma in my testing. Also in regards to the last code nulling "p" can i do something like this:

counter = 0; //initialise the counter
while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   storage[counter] = str; //use the counter as an index to add each value to the array
   counter++; //increment the counter
}
p = NULL; //clear the serial data storage outside of the loop after all the data has been retrieved
//using the liquid crystal library print out info
//kind of coding from memory so imagine that the cursor has been setup and everything
lcd.print("Speed: "); lcd.print(storage[0]); //print out the data stored in the value storage array index '0'
//set cursor to next line using appropiate commands again
lcd.print("Altitude: "); lcd.print(storage[1]); //ditto
// and so on and so on
//only thing i can figure now is to null the array by looping through and clearing it using a loop that repeats the number of times of "counter"

And that would be inserted in the "time to parse" section of the previous code. Does that seem like it could work reasonably? I am at work so i don't have my arduino with me so i'll have to test when i get back home. for now i'm trying to get the logic in order. Once again i really appreciate your help. This has given me a huge boost in getting this project working.

The strtok function takes a pointer to the string to parse, or a NULL to keep parsing the same string. You need to move the assignment of NULL to p INSIDE the curly braces, so that p is NULL for the next call.

Ah okay that explains it. Other than that however is the code more or less workable? i.e.

counter = 0; //initialise the counter
while ((str = strtok_r(p, ",", &p)) != "\n") // delimiter is the comma
{
   storage[counter] = str; //use the counter as an index to add each value to the array
   counter++; //increment the counter
   p = NULL; 
}

//using the liquid crystal library print out info
//kind of coding from memory so imagine that the cursor has been setup and everything
lcd.print("Speed: "); lcd.print(storage[0]); //print out the data stored in the value storage array index '0'
//set cursor to next line using appropiate commands again
lcd.print("Altitude: "); lcd.print(storage[1]); //ditto
// and so on and so on
//only thing i can figure now is to null the array by looping through and clearing it using a loop that repeats the number of times of "counter"

Other than that however is the code more or less workable?

Only one way to really know. Upload and see what happens.

Heh that's true. Well thanks again for the help .... i'll let you guys know how it turned out when i get back.

Well i'm probably gonna get home late tonight so i might as well post the experimental code.

NOTE THIS CODE IS UNTESTED. I am just posting this to get any feedback before i run it:

#include <LiquidCrystal.h>
#include <string.h>
char inData[80];
char flightdata[80];
byte index = 0;
LiquidCrystal lcd(12, 11, 2, 3, 4, 5);

void setup()
{
Serial.begin(9600);
lcd.begin(20, 2);
}

void loop()
{
   while(Serial.available() > 0)
   {
      char aChar = Serial.read();
      if(aChar == '\n')
      {
         // End of record detected. Time to parse
            char *p = inData; //assign the string to *p
            char *str;        //intialize str
            int counter = 0; //initialise the counter
            
            while ((str = strtok_r(p, ",", &p)) != "\0") // delimiter is the comma. NULL is the terminator
            {
                  flightdata[counter] = *str; //use the counter as an index to add each value to the array
                  counter++; //increment the counter

               p = NULL;
            }
            lcd.print("Speed: "); lcd.print(flightdata[0]); //print out the data stored in the value storage array index '0'
            //set cursor to next line using appropiate commands again
            lcd.print("Altitude: "); lcd.print(flightdata[1]); //ditto



         index = 0;
         inData[index] = NULL;
           
      }
      else
      {
         inData[index] = aChar;
         index++;
         inData[index] = '\0'; // Keep the string NULL terminated
      }
   }
}

Let me know if i overlooked anything ..... it compiles okay but that doesn't mean anything.

            char *p = inData; //assign the string to *p
            char *str;        //intialize str
            int counter = 0; //initialise the counter
            
            while ((str = strtok_r(p, ",", &p)) != "\0") // delimiter is the comma. NULL is the terminator

Some of the comments are wrong. You are not assigning the string to *p. You are making p point to the string.

You are not initializing str. You are declaring it.

These are trivial. There is no reason to have the != "\0" stuff in there. The str variable will point to a string or will point to NULL. Any non-NULL pointer will be true.

while (str = strtok_r(p, ",", &p))
{
}

ah good point,

I'll make the edits and i'll try to stop being lazy when i comment .... believe me i understand the value of accurate commenting .... especially when you write a 300 line piece of code, go away for two months and then try to remember what you did ^_^;;

Well i tried the script and it was a bust so i broke it down into smaller and smaller elements until all i had was this

#include <LiquidCrystal.h>

// initialize the library with the numbers of the interface pins
LiquidCrystal lcd(7, 8, 9, 10, 11, 12);

void setup() {
  // set up the LCD's number of rows and columns: 
  lcd.begin(20, 4);
}

void loop() {
  while(Serial.available() > 0)
   {
      
  lcd.setCursor(0, 0);
  lcd.print("LAT:XXXXXXXXXXXXXXXX");
  lcd.setCursor(0, 1);
  lcd.print("LON:XXXXXXXXXXXXXXXX");
  lcd.setCursor(0, 2);
  lcd.print("SPEED:");
  lcd.print(millis()/1000);
  lcd.setCursor(13, 2);
  lcd.print("HDG:");
  lcd.print(millis()/1000);
  lcd.setCursor(0, 3);
  lcd.print("ALT:XXXXXXYYXXXXXXXX");
  // print the number of seconds since reset:
  delay(1000);
  
   }
  
}

no matter what the screen just stays blank. I even tried adding in a delay just in case everything was happening too quickly. Any ideas what i should try now?

If the screen remains blank with this sketch, there are two possible causes. The first is that there is nothing sending the Arduino serial data. The second is that LCD is not working.

I presume you have verified that the LCD works.

If the LCD works, but no serial data is being received, it is not an Arduino software problem. The Arduino can not make serial data appear. The issue then is either a hardware problem or a software problem with the application that is sending the data or a communication problem (the Arduino and the sender are not getting exclusive access to the serial port).

Thanks again PaulS,

I did verify the lcd again with another sketch. And the only way i can be sure it is receiving serial data is the rx light blinking. The program sending the serial data may be overwhelming the arduino though with the amount of data it is sending. Is that possible?

The program sending the serial data may be overwhelming the arduino though with the amount of data it is sending. Is that possible?

Not really. The Arduino has a serial receive buffer that may fill up. The extra data is silently discarded, though.

If the rx light is blinking, the Arduino should be receiving data. However, I just noticed that you do not have a Serial.begin statement in setup() of the last sketch you posted. That would explain why the Arduino never gets the serial data.

Ah you are right. But in the initial sketch i did have it but no luck. That sketch is here:

#include <LiquidCrystal.h>
#include <string.h>
char inData[80];
char flightdata[80];
byte index = 0;
LiquidCrystal lcd(12, 11, 2, 3, 4, 5);

void setup()
{
Serial.begin(115200);
lcd.begin(20, 4);
}

void loop()
{
   while(Serial.available() > 0)
   {
      char aChar = Serial.read();
      if(aChar == '\n')
      {
            char *p = inData; //point to *p to the string in inData
            char *str;        //declaring *str
            int counter = 0; //initialise the counter

            while (str = strtok_r(p, ",", &p)) // delimiter is the comma
            {
                  flightdata[counter] = *str; //use the counter as an index to add each value to the array
                  counter++; //increment the counter

               p = NULL;
            }
            lcd.setCursor(0, 0);
            lcd.print("Speed: "); lcd.print(flightdata[0]); //print out the data stored in the value storage array index '0'
            //set cursor to next line using appropiate commands again
            lcd.setCursor(0, 1);
            lcd.print("Altitude: "); lcd.print(flightdata[1]); //ditto



         index = 0;
         inData[index] = NULL;
           
      }
      else
      {
         inData[index] = aChar;
         index++;
         inData[index] = '\0'; // Keep the string NULL terminated
      }
   }
}

This code does nothing until the incoming serial data contains a \n. Since nothing happens, I'm forced to assume that the incoming serial data does not actually contain a carriage return.

Just for giggles, change this:

if(aChar == '\n')

to this:

if(aChar == '\n' || index == 20)