Go Down

Topic: Getting Weather Data from XML with Ehernet Shield (Read 2069 times) previous topic - next topic

Hi,

I am trying to modify code from the cookbook, recipe 15.5 "Requesting Data From a Web Server Using XML" to get weather data off of Google's weather website:  http://www.google.com/ig/api?weather=Portland,OR

Under the <forecast_conditions> part of the website, I need the forecast high temp, low temp, and forecast condition.  For example <low data="55"/><high data="75"/> and <condition data="Chance of Rain"/>  The numbers 55 and 75 and the condition "Chance of Rain" will change every day, and I want these things sent to the serial monitor.  Eventually I will use them to control circuits.

Currently this code will print integers to the serial monitor but not char.

When I run the code, the serial monitor prints back:
Low is 55
High is 75
Condition is 0
Condition is .

It gives me random numbers and symbols for the Conditions.  Is this because they are not integers?  I am totally new to programing and have no idea how to fish characters or text out of the website.

Also...Because the forecast <condition data=x/> is the 2nd one listed on the website I used a counter to print both.  I actually don't want the first <condition data=x/> (which is today's condition) printed to the Serial monitor, but this is not my main problem.

Here is the code:
Code: [Select]

#include <Ethernet.h>
#include <TextFinder.h>
#include <SPI.h>

byte mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0x14, 0xBF }; //Ethernet Sheild MAC address
byte server[] = { 209,85,229,104 }; //Google XML weather page

EthernetClient client;

TextFinder finder(client);

char inString[32]; // string for incoming serial data
int stringPos = 0; // string index counter

int coldpin = 18;
int hotpin = 19;


void setup()
{
  Serial.begin(9600);
  Ethernet.begin(mac); 
  pinMode(coldpin,OUTPUT);
  pinMode(hotpin,OUTPUT);
 
}

void loop(){
  if (client.connect(server,80)) {
    client.println("GET /ig/api?weather=Portland,OR HTTP/1.0");
    client.println();
  }
  else
  {
    Serial.println("connection failed");
  }
 
  if (client.connected())
  {
    if(finder.find("<low data="))
    {
      int t_low = finder.getValue();
      Serial.print("Low is ");
      Serial.println(t_low);
           
      if(t_low <= 45){
        digitalWrite(coldpin, HIGH);
      }
    }
   
    if(finder.find("<high data="))
    {
      int t_high = finder.getValue();
      Serial.print("High is ");
      Serial.println(t_high);
     
      if(t_high >= 75)
      {
        digitalWrite(hotpin, HIGH);
      }
    }
    for (int i=0; i<2; i++){
    if(finder.find("<condition data="))  //THIS IS WHERE IT DOESN'T WORK (I think)
      {
        char inString[32]; // string for incoming serial data  ??
        int stringPos = 0; // string index counter ??
        char condition = finder.getValue();  //I tried char instead of int, but its not that simple
        Serial.print("Condition is ");
        Serial.println(condition);
       
           
    }
    else
    Serial.print("Could not find condition field");
  }
}
  else{
    Serial.println("Disconnected");
  }
  client.stop();
  client.flush();
  delay(6000);
}


Please help!  Thank you!

PaulS

Code: [Select]
char inString[32]; // string for incoming serial data
        char inString[32]; // string for incoming serial data  ??

Local and global variables of the same name are rarely a good idea.

Code: [Select]
      int t_low = finder.getValue();
      int t_high = finder.getValue();
        char condition = finder.getValue();  //I tried char instead of int, but its not that simple

So, what TYPE does Finder::getValue() return? It can not possibly return both character and integer values.

Code: [Select]
    for (int i=0; i<2; i++){
    if(finder.find("<condition data="))  //THIS IS WHERE IT DOESN'T WORK (I think)
      {
    }
    else
    Serial.print("Could not find condition field");
  }

Why are you doing this twice?

Wouldn't it be nice if your position of the { and your use of indentation were consistent?

Thanks for looking at this.

I forgot to delete these lines from the code before I posted it:

Code: [Select]

char inString[32]; // string for incoming serial data
        int stringPos = 0; // string index counter


Finder::getValue() returns integers.  Can I modify it or create something else to return text?  I need the forecast high temperature (an int), the forecast low temperature (an int), and the forecast condition i.e. "Rain" (text).  Would this be a char array?

Here is the Textfinder.h code:
Code: [Select]

/* 
  TextFinder.h - a simple text parser for Arduino data streams
  Copyright (c) 2009 2010 Michael Margolis.  All right reserved.
 
  This library is free software; you can redistribute it and/or
  modify it under the terms of the GNU Lesser General Public
  License as published by the Free Software Foundation; either
  version 2.1 of the License, or (at your option) any later version.

  This library is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser General Public
  License along with this library; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
 
  Version 1.1 - 21 August 2010 : added conditional compile for Stream class
              - August 2011: added conditional compile for arduino 1.0   
*/ 


#ifndef TextFinder_h
#define TextFinder_h

#if ARDUINO <= 22
#include "WProgram.h"
#else
// for Arduino 1.0
#include <Arduino.h>
#include <inttypes.h>
#endif

#if ARDUINO < 19  // defined by Arduino build process to indicate version
// version that don't use the Stream class need to include HardwareSerial and Client
#include <HardwareSerial.h>
#include <Client.h>
#else
#include "Stream.h"
#endif

class TextFinder {
private:
#if ARDUINO < 19 
  Client *clientStream;
  HardwareSerial *serialStream;
#else
  Stream *streamer;
#endif
  unsigned long timeout;    // number of seconds to wait for the next char before aborting read
  unsigned long startMillis; // used for timeout measurement
  int myRead();             // private function to read from the stream  - changed to return int instead of char !!!

public:
  // constructor:
  // default timeout is 5 seconds 
#if ARDUINO < 19 
  TextFinder(Client &stream, int timeout = 5);          // Ethernet constructor
  TextFinder(HardwareSerial &stream, int timeout = 5);  // Serial constructor
#else
  // this version is for Arduino versions that have Serial and Ethernet derived from stream
  TextFinder(Stream &stream, int timeout = 5);   
#endif 
  // find methods - these seek through the data but do not return anything
  // they are useful to skip past unwanted data 
  //
  boolean find(char *target);   // reads data from the stream until the target string is found
  // returns true if target string is found

  boolean findUntil(char *target, char *terminate);   // as above but search ends if the terminate string is found

 
  // get methods - these get a numeric value or string from the data stream 
  //
  long getValue();    // returns the first valid (long) integer value from the current position.
  // initial characters that are not digits (or the minus sign) are skipped
  // integer is terminated by the first character that is not a digit.

  long getValue(char skipChar); // as above but the given skipChar is ignored
  // this allows format characters (typically commas) in values to be ignored

  float getFloat();  // float version of getValue
  float getFloat(char skipChar);  // as above but the given skipChar is ignored
   
  int getString( char *pre_string, char *post_string, char *buffer, int length); //puts string found between given delimiters in buffer
  // string will be truncated to fit the buffer length
  // end of string determined by a match of a character to the first char of close delimiter
  // returns the number of characters placed in the buffer (0 means no valid data found)

  };
#endif


Thank you!

zoomkat

Code posted by somebody else in the past that might be worth looking at.

Code: [Select]

/*
* Based on SimpleClientGoogleWeatherDHCP - This one comes with High Temp and Current conditions data.
* gets xml data from http://www.google.com/ig/api?weather=milwaukee,wi
* reads temperature from field:  <temp_f data="66" />
* writes temperature  to 20x4 LCD. Using Ardiuno Mega 2640
*/

#if ARDUINO > 18
#include <SPI.h>         // needed for Arduino versions later than 0018
#endif

#include <Ethernet.h>
#include <EthernetDHCP.h>  // uses DHCP code from: http://blog.jordanterrell.com/post/Arduino-DHCP-Library-Version-04.aspx
#include <TextFinder.h>
#include <LiquidCrystal.h>
LiquidCrystal lcd(26, 24, 22, 23, 25, 27, 29);

int backLight = 28;    // pin 28 will control the backlight
byte mac[] = {
  0x90, 0xA2, 0xDA, 0x00, 0x8F, 0xD3 };
byte server[] = {
  209,85,229,104 }; // Google

Client client(server, 80);

TextFinder  finder( client ); 
char cond[40];
void setup()
{
  pinMode(backLight, OUTPUT);
  digitalWrite(backLight, HIGH); // turn backlight on. Replace 'HIGH' with 'LOW' to turn it off.
  lcd.begin(20,4);
  lcd.setCursor(0,0);
  lcd.print("I'm alive");
  if(EthernetDHCP.begin(mac)) {
    // begin method returns 1 if successful 
    lcd.setCursor (0,1);
    lcd.print("got IP address, connecting...");
    lcd.clear();
    delay(500); 
  }
  else {
    lcd.clear();
    lcd.print("unable to acquire ip address!");
    while(true)
      ;  // do nothing
  }
}


void loop()
{
  if (client.connect()) {
    client.println("GET http://www.google.com/ig/api?weather=milwaukee,wi");  // google weather for Milwaukee
    client.println();
  }
  else {
    lcd.println("Can not connect to Google Weather");

  }
  if (client.connected()) {
    finder.find("<temp_f data=");
    int temperature = finder.getValue();
    lcd.clear();
    lcd.setCursor(0,0);
    lcd.print("Temperature = ");
    lcd.print(temperature);
    delay(3000);
    lcd.setCursor(0,1);
    lcd.print("High Temp ");
    finder.find("<high data=");
    int temp = finder.getValue();
    lcd.print(temp);
    delay(3000);
    lcd.setCursor(0,2);
    lcd.print("Current Conditions ");
    finder.find("<condition data");
    finder.getString("=","/",cond,(40));
    lcd.setCursor(0,3);
    lcd.print(cond);

  }
  else {
    lcd.clear();
    lcd.print("Can not find xml tag");

  }
  client.stop();
  client.flush(); 
  delay(4000); // wait a minute before next update
}
Google forum search: Use Google Advanced Search and use Http://forum.arduino.cc/index in the "site or domain:" box.

That's It!  Thank you!  This forum rocks!

I have another question about how this revised code works.  Before (in my first post), I used a counter to print the first 2 times it found <condition data=, because I want to print the second time <condition data=> appears on the website, which is under <forecast_conditions>.  I changed it to look for <forecast_conditions> with an if statement and then look for <condition data= so that it will print the first forecast condition. 

But for some reason this code is printing the 2nd <condition data= after it finds <forecast_conditions> instead of the first one.  Anyone know what is going on???  Thank you!

Here is my revised code:
Code: [Select]

//Sources
//Arduino Cookbook 15.5,
//Google weather site http://www.google.com/ig/api?weather=Portland,OR
//wwww.bildr.org/?s=ethernet

#include <Ethernet.h>
#include <TextFinder.h>
#include <SPI.h>

byte mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0x14, 0xBF }; //Ethernet Sheild MAC address
byte server[] = { 209,85,229,104 }; //Google XML weather page

EthernetClient client;

TextFinder finder(client);

int coldpin = 18;
int hotpin = 19;

char cond[30]; // string for incoming serial data

void setup()
{
  Serial.begin(9600);
  Ethernet.begin(mac); 
  pinMode(coldpin,OUTPUT);
  pinMode(hotpin,OUTPUT);
}

void loop()
{
  if (client.connect(server,80))
  {
    client.println("GET /ig/api?weather=Portland,OR HTTP/1.0");
    client.println();
  }
 
  else
  {
    Serial.println("connection failed");
  }
 
  if (client.connected())
  {
    if(finder.find("<low data="))
    {
      int t_low = finder.getValue();
      Serial.print("Low Temp is ");
      Serial.println(t_low);
           
      if(t_low <= 45)
      {
        digitalWrite(coldpin, HIGH);
      }
    }
   
    if(finder.find("<high data="))
    {
      int t_high = finder.getValue();
      Serial.print("High Temp is ");
      Serial.println(t_high);
     
      if(t_high >= 75)
      {
        digitalWrite(hotpin, HIGH);
      }
    }
   
    if(finder.find("<forecast_conditions"))
    {
   
      if(finder.find("<condition data"))
      {
       finder.getString("=", "/", cond, (30));
       Serial.print("Forecast is ");
       Serial.println(cond);
       }
       
    else
    Serial.print("Could not find condition field");
    }
  }
 
  else
  {
    Serial.println("Disconnected");
  }
  client.stop();
  client.flush();
  delay(6000);
}

PaulS

I'd recommend that you use the Tools + Auto Format menu item to properly indent your code.

Code: [Select]
       finder.getString("=", "/", cond, (30));
Why is 30 in parentheses?

The TextFinder class has some advantages, but not being able to see what is being parsed is a huge drawback in my opinion.

The data being parsed may not be what you think it is.

Ahh...I see.  Thanks for the heads up on the Auto Format PaulS.

The 30 is in parentheses because that's how I copied it from the code that zoomkat posted.  I told you I was new and I don't know what I'm doing!  :)

So is there a way to see what is being parsed?  Or is there something else I can use besides TextFinder?



zoomkat

More client weather code for study:

Code: [Select]

// Include description files for other libraries used (if any)
#include <SPI.h>
#include <Ethernet.h>

// Define Constants
// Max string length may have to be adjusted depending on data to be extracted
#define MAX_STRING_LEN  20

// Setup vars
char tagStr[MAX_STRING_LEN] = "";
char dataStr[MAX_STRING_LEN] = "";
char tmpStr[MAX_STRING_LEN] = "";
char endTag[3] = {'<', '/', '\0'};
int len;

// Flags to differentiate XML tags from document elements (ie. data)
boolean tagFlag = false;
boolean dataFlag = false;

// Ethernet vars
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 113, 200 }; // www.weather.gov

// Start ethernet client
Client client(server, 80);

void setup()
{
  Serial.begin(9600);
  Serial.println("Starting WebWx");
  Serial.println("connecting...");
  Ethernet.begin(mac, ip);
  delay(1000);

  if (client.connect()) {
    Serial.println("connected");
    client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");   
    client.println();
    delay(2000);
  } else {
    Serial.println("connection failed");
  } 
}

void loop() {

  // Read serial data in from web:
  while (client.available()) {
    serialEvent();
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("Disconnected");
    Serial.println("==================================");
    Serial.println("");
    client.stop();

    // Time until next update
    //Serial.println("Waiting");
    for (int t = 1; t <= 15; t++) {
      delay(60000); // 1 minute
    }

    if (client.connect()) {
      //Serial.println("Reconnected");
      client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");   
      client.println();
      delay(2000);
    } else {
      Serial.println("Reconnect failed");
    }     
  }
}

// Process each char from web
void serialEvent() {

   // Read a char
char inChar = client.read();
   //Serial.print(".");
 
   if (inChar == '<') {
      addChar(inChar, tmpStr);
      tagFlag = true;
      dataFlag = false;

   } else if (inChar == '>') {
      addChar(inChar, tmpStr);

      if (tagFlag) {     
         strncpy(tagStr, tmpStr, strlen(tmpStr)+1);
      }

      // Clear tmp
      clearStr(tmpStr);

      tagFlag = false;
      dataFlag = true;     
     
   } else if (inChar != 10) {
      if (tagFlag) {
         // Add tag char to string
         addChar(inChar, tmpStr);

         // Check for </XML> end tag, ignore it
         if ( tagFlag && strcmp(tmpStr, endTag) == 0 ) {
            clearStr(tmpStr);
            tagFlag = false;
            dataFlag = false;
         }
      }
     
      if (dataFlag) {
         // Add data char to string
         addChar(inChar, dataStr);
      }
   } 
 
   // If a LF, process the line
   if (inChar == 10 ) {

/*
      Serial.print("tagStr: ");
      Serial.println(tagStr);
      Serial.print("dataStr: ");
      Serial.println(dataStr);
*/

      // Find specific tags and print data
      if (matchTag("<temp_f>")) {
      Serial.print("Temp: ");
         Serial.print(dataStr);
      }
      if (matchTag("<relative_humidity>")) {
      Serial.print(", Humidity: ");
         Serial.print(dataStr);
      }
      if (matchTag("<pressure_in>")) {
      Serial.print(", Pressure: ");
         Serial.print(dataStr);
         Serial.println("");
      }

      // Clear all strings
      clearStr(tmpStr);
      clearStr(tagStr);
      clearStr(dataStr);

      // Clear Flags
      tagFlag = false;
      dataFlag = false;
   }
}

/////////////////////
// Other Functions //
/////////////////////

// Function to clear a string
void clearStr (char* str) {
   int len = strlen(str);
   for (int c = 0; c < len; c++) {
      str[c] = 0;
   }
}

//Function to add a char to a string and check its length
void addChar (char ch, char* str) {
   char *tagMsg  = "<TRUNCATED_TAG>";
   char *dataMsg = "-TRUNCATED_DATA-";

   // Check the max size of the string to make sure it doesn't grow too
   // big.  If string is beyond MAX_STRING_LEN assume it is unimportant
   // and replace it with a warning message.
   if (strlen(str) > MAX_STRING_LEN - 2) {
      if (tagFlag) {
         clearStr(tagStr);
         strcpy(tagStr,tagMsg);
      }
      if (dataFlag) {
         clearStr(dataStr);
         strcpy(dataStr,dataMsg);
      }

      // Clear the temp buffer and flags to stop current processing
      clearStr(tmpStr);
      tagFlag = false;
      dataFlag = false;

   } else {
      // Add char to string
      str[strlen(str)] = ch;
   }
}

// Function to check the current tag for a specific string
boolean matchTag (char* searchTag) {
   if ( strcmp(tagStr, searchTag) == 0 ) {
      return true;
   } else {
      return false;
   }
}


Google forum search: Use Google Advanced Search and use Http://forum.arduino.cc/index in the "site or domain:" box.

I got the code to work!  Yay!  Thanks for your help zoomkat!  This code reads the next day's forecast off of Google's XML weather website and prints it to an LCD and the Serial Monitor.

Code: [Select]

//Sources:
//Arduino Cookbook 15.5,
//Google weather site http://www.google.com/ig/api?weather=Portland,OR
//wwww.bildr.org/?s=ethernet

#include <Ethernet.h>
#include <TextFinder.h>
#include <SPI.h>
#include <LiquidCrystal.h>
LiquidCrystal lcd(2, 3, 4, 5, 6, 7);

byte mac[] = { 0x90, 0xA2, 0xDA, 0x0D, 0x14, 0xBF }; //Ethernet Shield MAC address
byte server[] = { 209,85,229,104 }; //Google XML weather page
EthernetClient client;

TextFinder finder(client);

char cond[30]; // string for incoming serial data

void setup()
{
  Serial.begin(9600);
  Ethernet.begin(mac);
  lcd.begin(20,4);
}

void loop()
{
  if (client.connect(server,80))
  {
    client.println("GET /ig/api?weather=Portland,OR HTTP/1.0");
    client.println();
  }

  else
  {
    Serial.println("connection failed");
  }

  if (client.connected())
  {
    if(finder.find("<low data="))
    {
      int t_low = finder.getValue();
      Serial.print("Low Temp is ");
      Serial.println(t_low);

      lcd.setCursor(0,1);
      lcd.print("Low: ");
      lcd.setCursor(5,1);
      lcd.print(t_low);
      delay(1);
      lcd.setCursor(8,1);
      lcd.print((char)223);
      delay(1);
    }

    if(finder.find("<high data="))
    {
      int t_high = finder.getValue();
      Serial.print("High Temp is ");
      Serial.println(t_high);

      lcd.setCursor(0,0);
      lcd.print("High: ");
      lcd.setCursor(6,0);
      lcd.print(t_high);
      delay(1);
      lcd.setCursor(9,0);
      lcd.print((char)223);
      delay(1);
    }

    if(finder.find("<condition data"))
    {
      finder.getString("=", "/", cond, 30);
      Serial.print("Forecast is ");
      Serial.println(cond);

      lcd.setCursor(0,2);
      lcd.print(cond);
      delay(1);
    }

    else
      Serial.print("Could not find condition field");
  }

  else
  {
    Serial.println("Disconnected");
  }

  client.stop();
  client.flush();
  delay(6000);
}

Delovar

Hello!
Now I try to write the code for your own parser weather based Ethernet-Schild + Arduino.
My project is different from the one shown here: GET requests will be sent to multiple sites (4 or 5)) with the weather forecast.

Unfortunately, on August 27 service Weather Google Api no works.
Give advice: where else can you get the weather forecast for XML format for my project?

1. Weather.gov  GET /xml/current_obs/XXXX.xml HTTP/1.0
2.
3.
The more we do, the happier our future

Go Up
 


Please enter a valid email to subscribe

Confirm your email address

We need to confirm your email address.
To complete the subscription, please click the link in the email we just sent you.

Thank you for subscribing!

Arduino
via Egeo 16
Torino, 10131
Italy