Http request for xml data

Projects Programming
1

I have a roving networks wifly shield from a company called tinysine. I'm using the wiflyhq library from harlequin on github. I'm able to connect to my local wifi and send an http get request for a website like www.google.com and print it out from ... to tag.

When I replace that site for api.openweather.org/... along with its corresponding api call string, I get an error.

Before getting to the error, is there something special that needs to be done to an http request that returns xml data such as is the case here?

2

Include an Accept HTTP header

Accept: application/xml

It depends on the site, of course. For instance, you might need something like

Accept: application/weatherinfo+xml

Which would mean something like "I understand the XML transport for weatherinfo content".

3

When I replace that site for api.openweather.org/... along with its corresponding api call string, I get an error.

By now, surely you KNOW that "I get an error" without defining WHAT the error is AND posting your code is just wasting everyone's time.

4

Yes I do PaulS, thanks.

Thats why I included the "before getting to the error"... bit. I want to first make sure I understand how the http request should work so that I can then proceed to debug the error myself. If that fails then Ill come back and ask for help on the error itself along with the error.

Thats why my posts are so long...i always try to understand as much as possible of what is going on. :slight_smile:

5

Before getting to the error, is there something special that needs to be done to an http request that returns xml data such as is the case here?

No. While you may want to tell the server that you accept XML files, if you don't explicitly state that, the server will assume that you will accept them and treat them as text. Since that's what you will do, there is no reason to add to the overhead of communication with the server.

6

Ok, then Ill have to do more testing because when I download/print the google website, which is plain HTML, it prints fine onto the serial monitor, an api call which returns xml fails for some reason.

So far all I know about the error is that the WiFlyHQ library says: Failed to get prompt.

7

Ok here is what I have:

Sketch

#include <WiFlyHQ.h>

#include <SoftwareSerial.h>
SoftwareSerial wifiSerial(2,3);//originally8,9

WiFly wifly;

const char mySSID[] = "dlinkit";
const char myPassword[] = "mywifipwd";
//const char site[] = "www.google.com";
const char site[] = "api.openweathermap.org";
void terminal();

void setup(){
    char buf[32];

    Serial.begin(9600);
    Serial.println("Starting");
    Serial.print("Free memory: ");
    Serial.println(wifly.getFreeMemory(),DEC);

    wifiSerial.begin(9600);
    if (!wifly.begin(&wifiSerial, &Serial)) {
        Serial.println("Failed to start wifly");
  terminal();
    }

    /* Join wifi network if not already associated */
    if (!wifly.isAssociated()) {
  /* Setup the WiFly to connect to a wifi network */
  Serial.println("Joining network");
  wifly.setSSID(mySSID);
  wifly.setPassphrase(myPassword);
  wifly.enableDHCP();

  if (wifly.join()) {
      Serial.println("Joined wifi network");
  } else {
      Serial.println("Failed to join wifi network");
      terminal();
  }
    } else {
        Serial.println("Already joined network");
    }

    //terminal();

    Serial.print("MAC: ");
    Serial.println(wifly.getMAC(buf, sizeof(buf)));
    Serial.print("IP: ");
    Serial.println(wifly.getIP(buf, sizeof(buf)));
    Serial.print("Netmask: ");
    Serial.println(wifly.getNetmask(buf, sizeof(buf)));
    Serial.print("Gateway: ");
    Serial.println(wifly.getGateway(buf, sizeof(buf)));

    wifly.setDeviceID("Wifly-WebClient2");
    Serial.print("DeviceID: ");
    Serial.println(wifly.getDeviceID(buf, sizeof(buf)));

    if (wifly.isConnected()) {
        Serial.println("Old connection active. Closing");
        wifly.close();
    }

  callSite();
}

void callSite(){
  if (wifly.open(site, 80)) {
        Serial.print("Connected to ");
        Serial.println(site);
  
        /* Send the request */
        wifly.println("Accept: application/xml" );
        wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=mykey HTTP/1.0");
        //wifly.println("GET / HTTP/1.0");
        wifly.println();
  } else {
        Serial.println("Failed to connect");
    }
}

void loop(){
    if (wifly.available() > 0) {
  char ch = wifly.read();
  Serial.write(ch);
  if (ch == '\n') {
      /* add a carriage return */ 
      Serial.write('\r');
  }
    }

    if (Serial.available() > 0) {
  wifly.write(Serial.read());
    }
}

/* Connect the WiFly serial to the serial monitor. */
void terminal(){
    while (1) {
  if (wifly.available() > 0) {
      Serial.write(wifly.read());
  }
  if (Serial.available() > 0) {
      wifly.write(Serial.read());
  }
    }
}

I get these errors:

Starting
Free memory: 944
setPrompt hasnt been called
Already joined network
MAC: 00:06:66:6f:33:a1
IP: 192.168.1.55
Netmask: 255.255.255.0
Gateway: 192.168.1.1
DeviceID: Wifly-WebClient2
open api.openweathermap.org 80
Connected to api.openweathermap.org
HTTP/1.1 400 Bad Request
Server: openresty
Date: Wed, 11 May 2016 22:27:48 GMT
Content-Type: text/html
Content-Length: 166
Connection: close

400 Bad Request

400 Bad Request

nginx
8

HTTP/1.1 400 Bad Request

Hard to get much plainer than that.

http://openweathermap.org/data/2.5/forecast?q=Miami,us&mode=xml&appid=mykey
When I paste that into the address bar, I get:

{"cod": 401,"message": "Invalid API key. Please see Frequently Asked Questions - OpenWeatherMap for more info."}

The api key is NOT "mykey". It is the value stored in the variable named mykey.

9

Yes I know, I removed it from the code. If you want the key its: 709ad769a8432a1a909fa58a22d86b91

10

Ok i think I got it. I stumbled onto this:

where somewhere near the bottom Greg Meyer 19 Mar 2015 mentions:

In the new one you've posted, you added another println: client.print("Content-Length: "); # This used to be print rather than println client.println(data.length()); client.println(); client.println(data); So you were no longer malforming the requests.
[\quote]

So I changed my code to:

void callSite(){

if (wifly.open(site, 80)) {
       Serial.print("Connected to ");
       Serial.println(site);
 
       /* Send the request */
       //wifly.println("Accept: text/html" );
       wifly.println();
       wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=709ad769a8432a1a909fa58a22d86b91 HTTP/1.0");
       wifly.println();
       wifly.println("Host: api.openweathermap.org");
        //wifly.println("GET / HTTP/1.0");
       wifly.println();
 } else {
       Serial.println("Failed to connect");
   }
}
[\code]

I haven't figured out which println did it, but I was so excited I had to post! :slight_smile:

11

Great! Now I just have to put that thing into an array!

Can anyone point me to the code for that? :slight_smile:

12

The GET or POST line should be the first line in the request, not the "Accept" parameter.

        wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=mykey HTTP/1.0");
        wifly.println("Accept: application/xml" );
        wifly.println();

And Greg Meyer's code from reply #9 above has a problem. Too many line feeds. It should be this.

        /* Send the request */
        wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=709ad769a8432a1a909fa58a22d86b91 HTTP/1.0");
        wifly.println("Host: api.openweathermap.org");
        wifly.println();
13 
 wifly.println("Accept: application/xml" );
 wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=mykey HTTP/1.0");
 wifly.println();

SurferTim:
The GET or POST line should be the first line in the request, not the "Accept" parameter.

Yes. Another problem is that the OP is specifying HTTP/1.0 . HTTP version 1.0 doesn't take headers - it has to be a bare GET. Furthermore, I imagine that many web servers simply won't respond to 1.0 requests at all, these days.

Make this:

 wifly.println("GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=mykey HTTP/1.1");
 wifly.println("Accept: application/xml" );
 wifly.println();

In fact, you don't need the Accept line and that's not what the problem was. According to the spec:

If no Accept header field is present, then it is assumed that the client accepts all media types.

So try HTTP/1.1 on the request line. You don't need an Accept header, but you do need that terminating blank line.

14

Now I just have to put that thing into an array!

What thing?

char *apiKey = "709ad769a8432a1a909fa58a22d86b91";
char getStg[200];
sprintf(getStg, "GET /data/2.5/forecast?q=Miami,us&mode=xml&appid=%s HTTP/1.1", apiKey);
wifly.println(getStg);
15

Well what I get back is xml from openweather.org that looks like this...and I would like to put it into an array/dictionary so that I can parse the data Im interested in, out of it. For example, I just want the time dictionaries:

MiamiUS0.003

...I erased the ones in between but basically there are 8 of these per day for 5 days...

16

Marciokoko:
Well what I get back is xml from openweather.org that looks like this...and I would like to put it into an array/dictionary so that I can parse the data Im interested in, out of it. For example, I just want the time dictionaries:

There is no magic formula. You read the response on character at a time. After all the characters are read, parse it. That's what the WebClient example does.

17

ok thanks! I thought there might be an official library or a very popular library perhaps.

I did find this which is basically what you mention, parsing each character at a time:

http://forum.arduino.cc/index.php?topic=39023.0

And there is always the other option which I had thought of, and is mentioned in this post, having my own server fetch and serve a digested version of it to Arduino. I guess Id use php to fetch this xml, create an array, fetch the multiple day records and somehow average them (since I dont plan to have such a detailed forecast per 3 hours) and then serve that up for my arduino project to pick up in a much simpler format such as an array of 5 days, each day with just a dayID, temp, rain%, uv...

Oh I just noticed there is no UV or rain%, just humidity and pressure, so still, an array with 5 elements and each element dictionary with 4 keys (dayID, temp, humidity & psi).

18

I guess Id use php to fetch this xml, create an array, fetch the multiple day records and somehow average them (since I dont plan to have such a detailed forecast per 3 hours)

One php script should fetch the xml and create (and store) the data. That script should be running on the server, and should run again some time later.

Another php script that the Arduino calls, should get the records for the last n days, perform the averaging (by day?) and return just the last n days averages.

Push as much of the hard stuff off onto more powerful machines, whenever possible.

Of course, I can't really figure out what role the Arduino needs to have...

19

Thanks, the Arduino is part of a smart-home system that will give suggestions based on weather forecast :slight_smile: