Help parsing data with strtok

I’m retrieving data from a webserver, which is formatted like the following. They’re bus routes along with an upcoming estimated departure time in seconds from now. Each route can have up to 5 departures upcoming but that varies depending on time of day. I’m having some trouble understanding how to best parse the data in arduino. I’m used to writing in more flexible languages.

Here’s the string. (it contains new line ‘\n’ characters)

10s:984
8w:1331
43w:198
43w:846
43e:293
43e:907

In the end, I would like to have 4 arrays, one for each route, with the elements being the next departure time, with a place holder flag if there is no time (say ‘9999’ for arguments sake). The times can be negative too, to show that a bus has just left (up to -300). So after processing, it might look something like this:

int route10s[] = {984, 9999, 9999, 9999, 9999};
int route8w[] = {1331, 9999, 9999, 9999, 9999};
int route43w[] = {198, 846, 9999, 9999, 9999};
int route43e[] = {907, 9999, 9999, 9999, 9999};

Can anyone point me in the right direction for how to use strtok (or any other method) to do this?

Thanks!

strtok takes two parameters and returns a third.

The two parameters are a char * to the string you wish to tokenize, and a char * to the delimiting character (ie the separator). It returns a char * to the start of the token it finds. Using it to parse a string with multiple delimiters can be done like this:

char dataString[] = "10s:984\n8w:1331\n43w:198\n";
char * route;
char * departure;
char routeDelimiter = ':';
char departureDelimiter = '\n';

void setup(){
  Serial.begin(115200);
  Serial.println(dataString);
  route = strtok(dataString, &routeDelimiter);
  departure = strtok(NULL, &departureDelimiter);
  int departureInt = atoi(departure);
  printRoute(route, departureInt);
  while(1){
    route = strtok(NULL, &routeDelimiter);
    departure = strtok(NULL, &departureDelimiter);
    
    if(departure == NULL)
      break;
    int departureInt = atoi(departure);
    printRoute(route, departureInt);
  }
}

void loop(){
  

  
}

void printRoute(char * route, int departureInt){
  Serial.print("Route: ");
  Serial.print(route);
  Serial.print(", ");
  Serial.println(departureInt);
}

You can do it with regular expressions, see:

http://arduino.cc/forum/index.php?topic=59917

but how if my string take from other device such as ardu imu from serial data.

i want dataString=Serial.read();

thank’s

Serial.read() returns a single character from the serial stream. My blog contains a post covering how to read in serial strings from an external device (and in fact uses the Sparkfun Razor IMU for it’s example code)

I have been at this for hours. I found some code on an old forum and cannot locate it here. Anyways, I am trying to parse serial data (yeah I know its all over the place and I have looked all over the place). But, this code just loops and loops and I can’t get the component strings. The correct string gets to the while loop but it never terminates. Could someone please assist.

#include <string.h>

int cp = 0; // char pointer 
boolean cfound = false; // complete command found 
boolean cstart = false; // start of command
char stringIn[50]; // commandbuffer 
char parsed[50];   //parsed strings
int counter = 0;   //initialise the counter
byte index =0;

#define SOC '{' 
#define EOC '}'

void setup() {
 Serial.flush(); 
 Serial.begin(9600); 
 delay(500); 
 Serial.print("(:H:)"); 
}

 void loop() {
 int sa; 
 char bt; 
 sa = Serial.available(); // count serial buffer bytes
 
 if (sa > 0) { // if buffer not empty process buffer content
  for (int i=0; i < sa; i++){ 
   bt = Serial.read(); // read one char from the serial buffer
     
   if (cstart ) {
    stringIn[cp] = bt; 
    cp++;
     //Serial.print(bt);
   } 
   
   if (bt == SOC) { // got a start of command
    cstart = true;
    //Serial.println("start");
   }
     
   if (bt == EOC && cstart) { // check for last command char ) 
    cfound = true; 
    stringIn[--cp] = NULL;
    break; // end for-loop 
   } 
  } 
 } 
 
 if (cfound) { 
   for (int i=0; i<cp; i++){
      Serial.print(stringIn[i]);   //data is correct at this point
    }
    
 counter = 0; //initialise the counter
 char *p = stringIn; //assign the address of stringIn to *p
 char *str;          //intialize str
             
 while ((str = strtok_r(p, ":", &p)) != NULL) // delimiter is the colon
  {
   parsed[counter] = *str; //use the counter as an index to add each value to the array
   counter++; //increment the counter
   p = NULL;  
   Serial.print( parsed[counter]);
  }
  cp = 0;
  stringIn[cp] = NULL;
  cstart = false;
  cfound = false; 
  }
 }
 while ((str = strtok_r(p, ":", &p)) != NULL) // delimiter is the colon

Why? Why? Why?

There is an easy to use function called strtok(). There is a complex, hard to understand function called strtok_r().

The only advantage of the strtok_r() function over the strtok() function is that the the _r version is thread-safe. On the Arduino, is that really an advantage, considering that you only have one thread?

char *token = strtok(stringIn, ":");
if(token)
{
   parsed[counter++] = strdup(token); // You've got to COPY the data pointed to

   token = strtok(NULL, ":"); // Keep parsing the same string
   while(token)
   {
      parsed[counter++] = strdup(token); // You've got to COPY the data pointed to
      token = strtok(NULL, ":");
   }
}

This will end, and parsed[] will contain each of the tokens.

Or, it would except that this is wrong:

char parsed[50];   //parsed strings

Wrong. The parsed[] array is NOT an array of strings.

char *parsed[50]; would be an array of strings BUT it really is unlikely that you have enough memory, or that you are sending 49 colons in the input string. A more reasonable size is in order here.

Thanks much for your help. I pasted that code in and changed the array with *. I still get no strings out.

#include <string.h>

int cp = 0; // char pointer 
boolean cfound = false; // complete command found 
boolean cstart = false; // start of command
char stringIn[50]; // commandbuffer 
char *parsed[10];   //parsed strings
int counter = 0;   //initialise the counter
byte index =0;
char delims[]=":";

#define SOC '{' 
#define EOC '}'

void setup() {
 Serial.flush(); 
 Serial.begin(9600); 
 delay(500); 
 //Serial.print("(:H:)"); 
}

 void loop() {
 int sa; 
 char bt; 
 sa = Serial.available(); // count serial buffer bytes
 
 if (sa > 0) { // if buffer not empty process buffer content
  for (int i=0; i < sa; i++){ 
   bt = Serial.read(); // read one char from the serial buffer
     
   if (cstart ) {
    stringIn[cp] = bt; 
    cp++;
     //Serial.print(bt);
   } 
   
   if (bt == SOC) { // got a start of command
    cstart = true;
    //Serial.println("start");
   }
     
   if (bt == EOC && cstart) { // check for last command char ) 
    cfound = true; 
    stringIn[--cp] = NULL;
    break; // end for-loop 
   } 
  } 
 } 
 
 if (cfound) { 
   for (int i=0; i<cp; i++){
      Serial.print(stringIn[i]);   //data is correct at this point
    }
  
  counter = 0; //initialise the counter
  char *token = strtok(stringIn, ":");
  
  if(token)
  {
   parsed[counter++] = strdup(token); // You've got to COPY the data pointed to
   Serial.println( parsed[counter]);
   
   token = strtok(NULL, ":"); // Keep parsing the same string
   while(token)
   {
      parsed[counter++] = strdup(token); // You've got to COPY the data pointed to
      token = strtok(NULL, ":");
      Serial.println( parsed[counter]);  //nothing prints here
   }
   cp = 0;
   stringIn[cp] = NULL;
   cstart = false;
   cfound = false; 
   }
  }
 }

I think I found my issue. I am very new at this variable incrementing. Th print statements had nothing to print duh!! Thanks aging for the help.

Make sure you free all that memory that strdup() allocated as soon as you are done with it.