Parse and extract Data from pachube

Dear Friends, I'm using ethernet shield to connect to pachube to retrieve data.. This is serial monitor result after HTTP GET from pachube feed:

HTTP/1.0 200 OK
Date: Fri, 11 Feb 2011 11:08:06 GMT
Content-Type: text/plain; charset=utf-8
Last-Modified: Tue, 08 Feb 2011 13:56:36 GMT
Vary: Accept-Encoding
Connection: close

2,5,8,67

the total number of lines may change but my desired data is the last line . I want to store these received data (2,5,8,67) in separate variables...what is the solution for this? How can I reach to last line which then I can use sscanf to extract data ?

How is that data made to appear in the serial monitor? How that happens will influence how you parse the data.

If you are saving all the data in a String object, and then printing it, one method of parsing the data will work better.

If you are collecting the data in a char array, and then printing it, another method will work better.

If you are simply printing the characters as they arrive, parsing the data will be the least of your problems.

Thank you PaulS for reply..

How is that data made to appear in the serial monitor?

Actually as you mentioned, this is simply result of printing the characters as they arrive...

 if (client.available()) {
    char c = client.read();
    Serial.print(c);
     }

If you are simply printing the characters as they arrive, parsing the data will be the least of your problems.

So what is my first thing to do? store all received characters in an Array or string object?

If you are saving all the data in a String object, and then printing it, one method of parsing the data will work better.

Is this a true way : a) define a string object :

 String buffer = "";

b) add every characters which received via client.read() in to it :

buffer += c;

Or there is better(true) way?

Storing the data in a String object is the easiest to understand, but has a lot of overhead. It's a fine way to start, though.

Notice that you are only concerned about one record at a time. That is, every time that c is a carriage return ('\r') or line feed ('\n'), re-initialize the String object, instead of appending the character.

When there is no more data to read, the String object will contain only the last line.

The indexOf() and substring() methods will let you extract substrings ("2", "5", "8", and "67").

The toCharArray() method will extract the character array from the String object, and the atoi() function will convert that character array to an integer.

When there is no more data to read, the String object will contain only the last line.

How can I check that there is no more data to read? (only) by checking client.available() ?

The indexOf() and substring() methods will let you extract substrings ("2", "5", "8", and "67").

So, sscanf function is not useful for this problem?

(only) by checking client.available() ?

Yes.

So, sscanf function is not useful for this problem?

It could be. Keep in mind that sscanf wants a character array, not a String object. So, you must first extract the character array from the String object.

Below is some test code that downloads a very large data file from NOAA and extracts the latest wave period. I works by counting the line feeds at the end of the lines in the file downloaded, and when the desired line is reached, that line is captured. Then the desired data is extracted from the captured string using a string function. This probably could be improved, but it works as a starting point.

//zoomkat 12-22-10
//simple ethernet client test code
//for use with IDE 0021 and W5100 ethernet shield
//modify the arduino lan ip address as needed
//open serial monitor to see what the arduino receives
//push the shield reset button to run client again

#include <SPI.h>
#include <Ethernet.h>
String readString, readString1;
int x=0;
char lf=10;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 238, 27 }; // NOAA

Client client(server, 80);

void setup()
{
  Ethernet.begin(mac, ip);
  Serial.begin(9600);
  Serial.println("starting simple arduino client test");
  Serial.println();
  Serial.println("connecting...");

  if (client.connect()) {
    Serial.println("connected");
    client.println("GET /data/5day2/44013_5day.txt HTTP/1.0");
    client.println();
  } else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    //Serial.print(c);  // uncomment to see raw feed
    if (c==lf) x=(x+1);
    if (x==14) readString += c;
    }

  if (!client.connected()) {
     client.stop();

    Serial.println("Current data row:" );
    Serial.print(readString);
    Serial.println();
    readString1 = (readString.substring(41,43));
    Serial.println();
    Serial.print("DPD sec: ");
    Serial.println(readString1);
    Serial.println("done");

    for(;;);

    }
 }

Dear Zoomkat, thank you for your helpful code.. Firstly I've tested the code and got this result in Serial monitor: starting simple arduino client test

connecting...
connected
Current data row:

#YY  MM DD hh mm WDIR WSPD GST  WVHT   DPD   APD MWD   PRES  ATMP  WTMP  DEWP  VIS PTDY  TIDE

DPD sec: PD
done

I think you wanted to extract a number not words which is for you (Column DPD , ROW 14) = 3 I checked the code (view all received data in serial monitor) and found that my get return result has more 3 extra line So I added more 3 number by hand to variable x ( 17 instead of 14, which means parse 3 more line) :  if (x==17) readString += c;   and got this result :

starting simple arduino client test

connecting...
connected
Current data row:

2011 02 12 16 50 210  8.0  9.0   0.6     3   2.8  MM 1007.3   3.3   3.5  -6.7   MM -3.4    MM

DPD sec:  3
done

As I understand although your code is very useful but it's may not suitable in situation when return http result may change (like my test).. Is it true? How would you solve that?

Without knowing anything about your pachube page contains, I have no specific ideas. I suggest you look at the source of your page and locate something unique that you know it will contain, and which will just precede the data you want. Then you could write code that test the data being received for the unique item in the page, and start capturing the data at that point into a string.