Pages: [1]   Go Down
Author Topic: Reading html data only, storing as String  (Read 769 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This forum has been great at responding to question and I have yet another.

I not sure how to efficiently get the data I want using client.read() from a local webpage using a GET request

When i get the Data and view it over serial I get the following.


HTTP/1.1 200 OK
Server: Marvell 8688WM
Connection: close
Transfer-Encoding: chunked
Content-Type: application/json
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Pragma: no-cache

94
{"temp":68.00,"tmode":1,"fmode":0,"override":1,"hold":1,"t_heat":66.00,"tstate":0,"fstate":0,"time":{"day":5,"hour":23,"minute":20},"t_type_post":0}
0



I am only interested in the info after /n,  between "94"  &  "0".  More specific I really only need the int's from that section. as long as their in order I would know which was which.

So from the about html responce I would love to get it into a String variable like this:

68.00, 1, 0, 1, 1, 66.00, 0, 0, 5, 23, 20, 0

I did find some examples of client.read(), however it takes a minute or more to finish reading the values this way. Here is my test code:


Code:
void loop()
{
  
 if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
        readString += c;
        }  
    }
}
 
 if (count < 3){
    for (int i=1; i < 2; i++)  {
      Serial.print(readString);
      client.stop();
    }
    count = 4;
  }
}

Is their a better, faster, more efficient way of targeting this data?
Logged

UK
Offline Offline
God Member
*****
Karma: 17
Posts: 568
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

If using an Uno, remember you only have 2kB total SRAM. Strings eat RAM quickly, so storing the whole page in SRAM isn't a good idea.

Why not read each character 'c' and compare to the start of what you are looking for. Once you find that character then start to store in the string until you reach the last character, at which point stop adding the read bytes to the string.

You might also consider the byte length of your stored string and whether converting them to numerical values, such as int, long, float, is more efficient than storing as strings. I.e 68.00 as a string takes 5 bytes. As a float it takes 4. If you can afford to drop the decimals then you could store in an int for 2 bytes. Even use a uint8_t for a single byte for a number up to 255.

Is that data from an external page or one you generate yourself? If the latter then you could format so you only have the data you want to retrieve. You can generate a web page without all the other text easy enough.

If it's an external page then maybe make a little web service that first processes the page to strip out the extraneous text, leaving just the data for your Arduino to read and collect.

EDIT: Back on a PC and edited. I hate iPhone auto correct making your post sound like a jibbering alcoholic!
« Last Edit: February 10, 2013, 05:55:22 pm by tack » Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 601
Posts: 48543
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Code:
    for (int i=1; i < 2; i++)  {
      Serial.print(readString);
      client.stop();
    }

How many times is this loop going to iterate? For loops that explicitly iterate once are pretty stupid, in my opinion.

Do you really need a 2 byte int to hold the value 1?
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks for your replies. Obviously I am a extreme newbie, not just with arduino but with programming anything. In fact today marks my 2 week anniversary  since I wrote my first "hello world" so you'll have to forgive my "stupid" decisions when it comes to efficiency. To me, getting the arduino to do something I want, efficient or not, is a big victory for me.

Quote
Is that data from an external page or one you generate yourself?

The web page is generated from my thermostat and I don't have any means of altering it. As far as i can tell the website is very simple, i don't think it even contains HTML as the source page looks identical to the page i see in the browser which contains only this:

{"temp":66.00,"tmode":1,"fmode":0,"override":1,"hold":1,"t_heat":66.00,"tstate":0,"fstate":0,"time":{"day":6,"hour":9,"minute":29},"t_type_post":0}

Quote
Why not read each character 'c' and compare to the start of what you are looking for. Once you find that charter the. Start to store in the string until you reach the last character, at which point stop adding the read bytes to the string.

This sounds like a great idea, I just don't know where to begin. Could you perhaps provide an example? or point me in the right direction.

Quote
How many times is this loop going to iterate? For loops that explicitly iterate once are pretty stupid, in my opinion.

 PaulS, ironically I spent a good 20 mins learning the "for" loop (so i didn't look dumb posting code that looped my client.read() over and over) thinking it was a great solution to having my Serial.print print only once. I guess avoiding looking stupid backfired.

I'm all ears on learning new and better ways of doing things.


Logged

New Jersey
Offline Offline
Faraday Member
**
Karma: 65
Posts: 3638
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You're going to need to look at null terminated arrays of char. Declare a global array and an index that tells you which cell to put the next character in. When you read a char from the client, put in in the next place in the array, increment the index and null terminate. Make sure you don't try to use more cells than you allocated. When you see a '\n', examine the array to see if it starts with '{'. If it does, parse out the data you're looking for. Search the forums for idx - you'll find plenty of examples of this technique, good and bad.
Logged

UK
Offline Offline
Shannon Member
****
Karma: 222
Posts: 12541
-
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

For such a simple parsing requirement I'd just process the input stream one character at a time, looking for the opening brace ('{').

Once found, process each subsequent character, probably in a switch statement:

If it's a decimal digit or a period, append to a buffer holding the current input field.
If it's a closing brace ('}'), the input line is complete. Save the content of the buffer (if any) and end the parsing.
If it's any other character, save the content of the buffer (if any) and reset the buffer ready for the next field.
Logged

I only provide help via the forum - please do not contact me for private consultancy.

0
Offline Offline
Tesla Member
***
Karma: 141
Posts: 9470
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

In the below string you already have the start of data marker colon : and end of data marker comma ,. There have been previously posted parsing code that uses start of packet and end of packet markers. That code could be added to to make a new string with commas added between the data packets as a new delimiter.

Code:
{"temp":66.00,"tmode":1,"fmode":0,"override":1,"hold":1,"t_heat":66.00,"tstate":0,"fstate":0,"time"
Logged

Consider the daffodil. And while you're doing that, I'll be over here, looking through your stuff.   smiley-cool

Temple, Texas
Offline Offline
Sr. Member
****
Karma: 14
Posts: 360
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

For such a simple parsing requirement I'd just process the input stream one character at a time, looking for the opening brace ('{').

Once found, process each subsequent character, probably in a switch statement:

If it's a decimal digit or a period, append to a buffer holding the current input field.
If it's a closing brace ('}'), the input line is complete. Save the content of the buffer (if any) and end the parsing.
If it's any other character, save the content of the buffer (if any) and reset the buffer ready for the next field.

I agree with this except it's a little bit harder, because he has nested curlies {,,,{,,,}}
Logged

Temple, Texas
Offline Offline
Sr. Member
****
Karma: 14
Posts: 360
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

In the below string you already have the start of data marker colon : and end of data marker comma ,.

There will also be these colons and commas that need to be "bypassed"  though:
Code:
Cache-Control: no-store, no-cache, must-revalidate
Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

For such a simple parsing requirement I'd just process the input stream one character at a time, looking for the opening brace ('{').

Once found, process each subsequent character, probably in a switch statement:

Not sure how to do this... not even sure where to begin looking.  here is how i think it might look (OBVIOUSLY WAY WRONG AS IT DOESN'T EVEN COMPILE).

Sorry for my ignorance, i'm trying folks.

Code:
void loop()
{
 
char start = "{";
 
 if (client) {
    while (client.connected()) {
      if (client.available()) {
       
            if (client.read() == start);
            {
              char c = client.read()
              readString += c;
            }
        }   
    }
}
 
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
    for(;;)
;
  }
}
Logged

Temple, Texas
Offline Offline
Sr. Member
****
Karma: 14
Posts: 360
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Let's say you are able to get it into a string like "68.00, 1, 0, 1, 1, 66.00, 0, 0, 5, 23, 20, 0".  What are you going to do with it then?



Logged

Offline Offline
Newbie
*
Karma: 0
Posts: 16
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Let's say you are able to get it into a string like "68.00, 1, 0, 1, 1, 66.00, 0, 0, 5, 23, 20, 0".  What are you going to do with it then?

Eventually have the arduino at as a server where i could query it for these settings.

This string would be converted into an array.
Logged

UK
Offline Offline
Shannon Member
****
Karma: 222
Posts: 12541
-
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'd say you were on the right lines, although it's obviously incomplete. I was envisaging something like this:

Code:
// incomplete, untested
while (client.connected())
{
  if (client.available())
  {
    char c;
    // throw away everything up to the first '{'
    do
    {
      c = client.read();
    }
    while (c != '{');

    char buffer[32+1];
    byte length = 0;
    boolean done = false;
    float values[32];
    byte valueCount = 0;
    while(!done)
    {
      c = client.read();
      switch(c)
      {
      case '\n':
      case '\r':
        // end of line
        if(length > 0)
        {
          // buffer contains the last field
          values[valueCount++] = atof(buffer);
        }
        length = 0;
        done = true;
        break;

      case '0' .. '9':
      case '.':
        if (length < 32)
        {
          buffer[length++] = c;
          buffer[length] = 0;
        }
        break;

      default:
        if(length > 0)
        {
          // buffer contains the previous field
          values[valueCount++] = atof(buffer);
        }
        length = 0;
      }
    }; // end while !done

    doSomethingWith(values, valueCount);
  } // end if client.available()
} // end while connected
Logged

I only provide help via the forum - please do not contact me for private consultancy.

0
Offline Offline
Tesla Member
***
Karma: 141
Posts: 9470
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

In the below string you already have the start of data marker colon : and end of data marker comma ,.

There will also be these colons and commas that need to be "bypassed"  though:
Code:
Cache-Control: no-store, no-cache, must-revalidate

Probably kind of crude, but one might count the carrage returns until the desired data line is reached. Below is code from a while back that extracts the desired wave period from a downloaded NOAA file that contains the desired data. The below link downloads the page to a browser to see what is there.

http://140.90.238.27/data/5day2/44013_5day.txt

Code:
//zoomkat 12-22-10
//simple ethernet client test code
//for use with IDE 0021 and W5100 ethernet shield
//modify the arduino lan ip address as needed
//open serial monitor to see what the arduino receives
//push the shield reset button to run client again

#include <SPI.h>
#include <Ethernet.h>
String readString, readString1;
int x=0;
char lf=10;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 238, 27 }; // NOAA

EthernetClient client; //(server, 80);

void setup()
{
  Ethernet.begin(mac, ip);
  Serial.begin(9600);
  Serial.println("starting simple arduino client test");
  Serial.println();
  Serial.println("connecting...");

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /data/5day2/44013_5day.txt HTTP/1.0");
    client.println();
  } else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    //Serial.print(c);  // uncomment to see raw feed
    if (c==lf) x=(x+1);
    if (x==14) readString += c;
    //readString += c;
  }

  if (!client.connected()) {
     client.stop();

    Serial.println("Current data row:" );
    Serial.print(readString);
    Serial.println();
    readString1 = (readString.substring(41,43));
    Serial.println();
    Serial.print("DPD sec: ");
    Serial.println(readString1);
    Serial.println("done");

    for(;;);

    }
 }
 
Logged

Consider the daffodil. And while you're doing that, I'll be over here, looking through your stuff.   smiley-cool

Pages: [1]   Go Up
Jump to: