Parsing comma separated string values

Hello!

I know this has been discussed before, but I can't find some useful examples to achieve my goal... although I am very close to success.

So, I have a weather station, that sends the data to wunderground.com ( that part is OK ) and also it sends data to a mysql database on my server for later use. Now I want to extract this data from mysql and use them on LCD display in my house.

With a help of some php scripts, that interacts with mysql database, I managed to make arduino extract the echo response from my webserver, wich is something like this:

-0.81,89.90,0.00,3.05,15.06,19.12,NNW,Gentle Breeze,

this means: temperature,humidity,rain1h,rain24h,windspeed,windgusts,winddirection,windstrenght).

Now I want to extract the information between comma's, in separate strings... or if they are values, in floats...whatever.

I managed to do that, but unfortunately, it does not work all the time. why?
Well I understand why, but I don't know the solution.

It happens that if some values change... Let's take for example windspeed: if it's 8.55 or 20.40 the parsing script isn't working the same way. Because the first value has four characters (8 . 5 5 ) and the second has five ( 2 0 . 4 0). So many is the times when I have wrong values, sometimes with comma included extracted from there.

I will post the script for extracting the server response (just for other people to see it, but the problem is not here) and most important the parsing script... which I know is problematic.

Any help would be appreciated.

Thank you!
Adrian.

void getwsdata()
  {

   if (client.connect(server, 80)) 
  {
  Serial.println("connected to server");
  // Make a HTTP request:
  link=String("GET http://"+serverip+location);
  link.toCharArray(buf,150);
  sprintf(buffer,"%s",buf,"HTTP/1.1");
  client.println(buffer);
  client.println();
  delay(500);
  }

    // if there are incoming bytes available 
  // from the server, read them and print them:
  Serial.print("Server echo:");
  while (client.available()) {
  c = client.read();
  Serial.print(c);
   databuffer += c;
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println("disconnecting from server.");
    client.println( "Connection: close" );
    client.println();
    client.println();
    client.stop();
    }
else
{
  Serial.println("Connection Error:");
}
Serial.println();
}
 
void processmysqldata()
{
      int firstCommaIndex = databuffer.indexOf(',');
      int secondCommaIndex = databuffer.indexOf(',', firstCommaIndex+1);
             
      String wstemp = databuffer.substring(2, firstCommaIndex);
      String wshum = databuffer.substring(firstCommaIndex+1, secondCommaIndex);
      String wsrain1h = databuffer.substring(secondCommaIndex+1, secondCommaIndex+5);
      String wsrain24h = databuffer.substring(secondCommaIndex+6, secondCommaIndex+10);
      String wsws = databuffer.substring(secondCommaIndex+11, secondCommaIndex+16);
      String wswg = databuffer.substring(secondCommaIndex+17, secondCommaIndex+22);
      String wswd = databuffer.substring(secondCommaIndex+23, secondCommaIndex+26);
      String wswp = databuffer.substring(secondCommaIndex+27, secondCommaIndex+42); 

     databuffer = "";
}

Receive your data into the data buffer as a character array terminated with A NULL. Then use the strtok function to parse the data with the commas as delimiters. The serial input basics thread may have useful information on receving and parsing the data (parsing, see example 5).
Avoid using the String class, if possible. The String class can cause memory problems if not used carefully.

1 Like

Thank you!

I followed the nr.5 example as you indicate, and I adapted the code for my situation. Seems that it is working fine. I have to wait for the wind to settle down, to see if the data parser is working correctly... no matter what the values between comma's are... so it has to work even is the wind is 0 or 18.95...

Thanks again!

Best regards,
Adrian

I wrote this a couple of years back to deal with GPS data.

void getCSVfields()
{
  byte sentencePos =0;
  commaCount=0;
  msgField[commaCount]="";
  while (sentencePos < fullMsg.length())
  {
    if(fullMsg.charAt(sentencePos) == ',')
    {
      commaCount++;
      msgField[commaCount]="";
      sentencePos++;
    }
    else
    {
      msgField[commaCount] += fullMsg.charAt(sentencePos);
      sentencePos++;
    }
  }

It assumes that the complete message is in fullMsg and that you’ve declared an array of strings called msgField.

DKWatson:
It assumes that the complete message is in fullMsg and that you've declared an array of strings called msgField.

It is not a good idea to use the String (capital S) class on an Arduino as it can cause memory corruption in the small memory on an Arduino. Just use cstrings - char arrays terminated with 0.

...R

you can use sscanf to "extract" the data:

    char s[] = "-0.81,89.90,0.00,3.05,15.06,19.12,NNW,Gentle Breeze,";

    float f1, f2, f3, f4, f5, f6;
    char d[30];

    int result = sscanf(s, "%f,%f,%f,%f,%f,%f,%s", &f1, &f2 ,&f3, &f4, &f5,&f6, d);

you must do something extra to split "NNW" and "Gentle Breeze"
"result" will have the number of data that is extracted: in this case it should be 7

Note: this only works if:

  1. "s" is a cstring (\0 terminated char array)
  2. the format must be exactly that: 1 float followed by a comma followed by a float ... (but you can change the format)
  3. must ensure that "d" is big enough to hold the last string

As others have said, Don‚Äôt Use String‚ĄĘ. It is especially dangerous when used with the larger Ethernet libraries. The only thing worse is an array of Strings. :-/

Here’s a little test program that shows how to use characater arrays (aka C strings) to format the request and how to parse a response:

//  These parts of the request never change, so store them in FLASH to save RAM
const char   action     [] PROGMEM = "GET http://";
const size_t actionLen             = sizeof(action) - 1; // don't count the zero byte at the end
const char   httpVersion[] PROGMEM = " HTTP/1.1"; // space to separate from ip+loc
const size_t httpVersionLen        = sizeof(httpVersion)-1;

//  These parts are *probably* variable.  I can't see the rest of your sketch :-/
char serverip[] = "123.123.123.123";
char location[] = "/what goes here?/";


//  Some variables to receive a line of characters
size_t         count     = 0;
const size_t   MAX_CHARS = 64;
char           line[ MAX_CHARS ];


//  Print an HTTP GET request to the specified output destination
void sendRequestTo( Print & output )  // could be Serial or ethernet client
{
  //  Calculate how big the request is going to be
  size_t ipLen      = strlen(serverip);
  size_t locLen     = strlen(location);
  size_t requestLen = actionLen + ipLen + locLen + httpVersionLen;

  //  Declare a local buffer to hold the entire request, plus a little.
  char buffer[ requestLen + 8 ];
  
  //  Copy all those pieces into the buffer
  char *ptr = &buffer[0];  // ptr points to the beginning of the char array.

  strcpy_P( ptr, action ); // start by copying in the action string
  ptr = &ptr[ actionLen ]; // point to the next place

  strcpy( ptr, serverip ); // "append" the IP addr
  ptr = &ptr[ ipLen ];     // point to the next place

  strcpy( ptr, location ); // "append" the location
  ptr = &ptr[ locLen ];    // point to the place

  strcpy_P( ptr, httpVersion );  // all done!

  output.println( buffer );
  output.println();
}


void setup()
{
  Serial.begin( 9600 );

  sendRequestTo( Serial ); // could have been client
}

void loop()
{
  if (lineReady( Serial )) {  // could have been client
    parseLine( line );
  }
}

//----------------------------------------------
//  Parse a line of text with this CSV format:
//    temperature,humidity,rain1h,rain24h,windspeed,windgusts,winddirection,windstrength
//  Example:
//    "-0.81,89.90,0.00,3.05,15.06,19.12,NNW,Gentle Breeze, "

void parseLine( char *line )
{
  char windDir     [  4 ]; // max 3 characters plus NUL terminator
  char windStrength[ 16 ]; // just a guess

  float temperature, humidity, rain1h, rain24h, windspeed, windgusts;
  char *field;
  
  field       = strtok( line, "," ); // start the process with line
  temperature = atof( field );
  
  field       = strtok( nullptr, "," ); // *continue* the process with nullptr
  humidity    = atof( field );

  field       = strtok( nullptr, "," );
  rain1h      = atof( field );

  field       = strtok( nullptr, "," );
  rain24h     = atof( field );

  field       = strtok( nullptr, "," );
  windspeed   = atof( field );

  field       = strtok( nullptr, "," );
  windgusts   = atof( field );

  field       = strtok( nullptr, "," );
  strncpy( windDir, field, sizeof(windDir)-1 );
      // copy *limited number* of characters with strNcpy (may not copy zero byte)
  windDir[ sizeof(windDir)-1 ] = '\0'; // make sure its NUL-terminated

  field       = strtok( nullptr, "," );
  strncpy( windStrength, field, sizeof(windStrength)-1 );
  windStrength[ sizeof(windStrength)-1 ] = '\0';

  //  If you need to test the wind direction/strength strings, this is how you compare
  if (strcmp_P( windDir, PSTR("NNW") ) == 0) {                // PSTR saves RAM here

    if (strcmp_P( windStrength, PSTR("Gentle Breeze") ) == 0) //   ... and here
      Serial.println( F("Nice") );                            // Here, use F to save RAM
    else if (strcmp_P( windStrength, PSTR("Gusty") ) == 0)
      Serial.println( F("'Tis an ill wind!") );
    else
      Serial.println( windStrength );
    
  } else {
    Serial.print( F("Winds out of the ") );
    Serial.println( windDir );
  }
  
  Serial.print( F("Temp is ") );
  Serial.println( temperature );

  Serial.print( F("Wind gusts of ") );
  Serial.println( windgusts );
}


// Gradually accumulate a line of text from the input stream.

bool lineReady( Stream & input )    // could be Serial or ethernet client
{
  bool          ready     = false;
  const char    endMarker = '\n';

  while (input.available()) {

    char c = input.read();

    if (c != endMarker) {
      // Only save the printable characters, if there's room
      if ((' ' <= c) and (count < MAX_CHARS-1)) {
        line[ count++ ] = c;
      }
    } else {
      //  It's the end marker, line is completely received
      line[count] = '\0'; // terminate the string
      count       = 0;    // reset for next time
      ready       = true;
      break;
    }
  }

  return ready;

} // lineReady

This sketch uses the Serial Input Basics technique to accumulate a line from the client (Serial in this example).

As GroundFungus suggested, it uses strtok to step through the fields, regardless of how many characters each comma-separated field has.

It also uses atof, strcpy and strcpy_P. Note the use of strncpy to make sure it doesn’t write past the end of character arrays. As mentioned above, this can be a problem with sscanf.

There are many others, so just ask if you need help.