Go Down

Topic: trouble using GET to retrieve public API XML data (Read 2587 times) previous topic - next topic

bobbyzacharias

Aug 19, 2014, 12:08 am Last Edit: Aug 19, 2014, 12:13 am by bobbyzacharias Reason: 1
Hello all,

I've done lots of searching and hours of trying to figure this out myself and am stuck. I'm working with some local high school students to make a machine that needs to know the local ocean tide height as an input.

NOAA has a very rich API that can provide me this data; the exact information I need (should) be visible to you if you load http://tidesandcurrents.noaa.gov/api/datagetter?date=latest&station=8516945&product=one_minute_water_level&datum=STND&units=metric&time_zone=gmt&application=web_services&format=xml
(The height data we need is tagged with "v". If you want to read NOAA's API documentation, head over to http://tidesandcurrents.noaa.gov/api/.)

I've modified the built-in "Web Client" example to request this data. Here is the code I'm running:

Code: [Select]
/*
 Web client modified to read wave height data

This sketch connects to a website (http://www.google.com)
using an Arduino Wiznet Ethernet shield.

Circuit:
* Ethernet shield attached to pins 10, 11, 12, 13

created 18 Dec 2009
by David A. Mellis
modified 9 Apr 2012
by Tom Igoe, based on work by Adrian McEwen

*/

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = { 0x90, 0xA2, 0xDA, 0x0F, 0x9C, 0xEB };
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(74,125,232,128);  // numeric IP for Google (no DNS)
char server[] = "http://tidesandcurrents.noaa.gov";    // name address for Google (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192,168,0,177);

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
// Open serial communications and wait for port to open:
 Serial.begin(9600);
  while (!Serial) {
   ; // wait for serial port to connect. Needed for Leonardo only
 }

 // start the Ethernet connection:
 if (Ethernet.begin(mac) == 0) {
   Serial.println("Failed to configure Ethernet using DHCP");
   // no point in carrying on, so do nothing forevermore:
   // try to congifure using IP address instead of DHCP:
   Ethernet.begin(mac, ip);
 }
 // give the Ethernet shield a second to initialize:
 delay(1000);
 Serial.println("connecting...");

 // if you get a connection, report back via serial:
 if (client.connect(server, 80)) {
   Serial.println("connected");
   // Make a HTTP request:
   client.println("GET /api/datagetter?date=latest&station=8516945&product=one_minute_water_level&datum=STND&units=metric&time_zone=gmt&application=web_services&format=xml HTTP/1.0");
   client.println("Host: tidesandcurrents.noaa.gov");
   client.println("Connection: close");
   client.println();
 }
 else {
   // kf you didn't get a connection to the server:
   Serial.println("connection failed");
 }
}

void loop()
{
 // if there are incoming bytes available
 // from the server, read them and print them:
 if (client.available()) {
   char c = client.read();
   Serial.print(c);
 }

// if the server's disconnected, stop the client:
 if (!client.connected()) {
   Serial.println();
   Serial.println("disconnecting.");
   client.stop();

   // do nothing forevermore:
   while(true);
 }
}


However, though I've confirmed that the Ethernet shield is online and functioning (running the original, unmodified version of this code works as it should), I get the following serial output when I run the above sketch:

Quote
connecting...
connected

disconnecting.


I've tried to modify the code posted in http://forum.arduino.cc/index.php?topic=106699.0 for the present purpose but 1) the Google API is long gone so I can't even use it as a test case and 2) regardless, even configuring that code to retrieve from NOAA, I don't get any data back.

Any help is greatly appreciated! I've been stumped with this all afternoon.

PaulS

Code: [Select]
char server[] = "http://tidesandcurrents.noaa.gov";    // name address for Google (using DNS)
Nonsense. That is not google name!

How long does it take from the end of sending the GET request to the end of setup()? How long does it take for loop() to start?

Compare that total time to how long it takes the server to start providing a response. Which is longer?

What happens when the server has not (yet) started to return a response by the time loop() starts?
The art of getting good answers lies in asking good questions.

SurferTim

The Arduino ethernet library is apparently having problems with the DNS on that domain. Use an IP address instead. It works fine here using an IP.
Code: [Select]
IPAddress server(140,90,78,215);


I even tried removing the protocol from your server entry and it still has trouble with resolving the dns on that domain.

bobbyzacharias

Paul S:

Thanks for your ideas! Also sorry for leaving that Google server comment, it was from the original sketch.

I implemented the timers as you asked (you can look at my code and tell me if the timers here aren't set up correctly) and running this code I get the output pasted below the code.
Code: [Select]
/*
  Web client modified to read wave height data

created 18 Dec 2009
by David A. Mellis
modified 9 Apr 2012
by Tom Igoe, based on work by Adrian McEwen

modified 8/19/14 by bobby zacharias

*/

#include <SPI.h>
#include <Ethernet.h>
int GETtimer;
int loopstart;

byte mac[] = { 0x90, 0xA2, 0xDA, 0x0F, 0x9C, 0xEB };
char server[] = "http://tidesandcurrents.noaa.gov";
IPAddress ip(140,90,78,215); //

EthernetClient client;

void setup() {
  Serial.begin(9600);
   while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }

  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    // try to congifure using IP address instead of DHCP:
    Ethernet.begin(mac, ip);
  }
  delay(1000);
  Serial.println("connecting...");

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /api/datagetter?date=latest&station=8516945&product=one_minute_water_level&datum=STND&units=metric&time_zone=gmt&application=web_services&format=xml HTTP/1.0");
    GETtimer = millis();
    client.println("Host: tidesandcurrents.noaa.gov");
    client.println("Connection: close");
    client.println();
  }
  else {
    Serial.println("connection failed");
  }
  GETtimer = millis()-GETtimer;
  Serial.print("GET to end of setup took "); Serial.print(GETtimer); Serial.println(" milliseconds");
}

void loop()
{
  loopstart = millis();
  Serial.print("loop start took "); Serial.print(loopstart); Serial.println(" milliseconds");
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
    while(true);
  }
}

Output from running the above 6 times in a row is:
Quote
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 5571 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1613 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.
connecting...
connected
GET to end of setup took 0 milliseconds
loop start took 1563 milliseconds

disconnecting.

I have no idea why the loop start time departed from 1563ms those two times. Also you asked me
Quote
Compare that total time to how long it takes the server to start providing a response. Which is longer?
but I don't know how to answer that because I don't know how to determine the server's timing. Is there some good way to do that?

As you point out, if the server hasn't yet started responding by the time loop() starts, it will see that client.connected fails and disconnect; I inserted a delay at the end of setup() to try to give the server time, thinking 50ms would be reasonable, but that didn't change anything. I also tried values between 100 and 500ms, still no change. I don't think inserting any extra waiting time is a good idea, though, because the xml page being requested is tiny--just 210 bytes--so it should transmit in its entirety pretty quick.

Let me know if you think of anything else to try?


SurferTim:

I did substitute in the IP address you looked up for me (thanks!) but it didn't change the output at all. In fact, if the DNS lookup was the problem, it seems like I should've gotten the serial message "Failed to configure Ethernet using DHCP" but I haven't seen that yet. Any other ideas?

SurferTim

It's not the DHCP that is failing. It is the DNS resolution on that domain. I'll post my working version of your code latter.

bobbyzacharias


SurferTim

#6
Aug 20, 2014, 01:28 pm Last Edit: Aug 20, 2014, 01:39 pm by SurferTim Reason: 1
There is a difference between a DHCP server's failure to issue a DNS server, and the DNS server issued being unable to resolve the domain name correctly.

This code works for me.
Code: [Select]
/*
 Web client modified to read wave height data

This sketch connects to a website (http://www.google.com)
using an Arduino Wiznet Ethernet shield.

Circuit:
* Ethernet shield attached to pins 10, 11, 12, 13

created 18 Dec 2009
by David A. Mellis
modified 9 Apr 2012
by Tom Igoe, based on work by Adrian McEwen

*/

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = { 0x90, 0xA2, 0xDA, 0x0F, 0x9C, 0xEB };
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
IPAddress server(140,90,78,215);
//char server[] = "co-ops-nos.1.noaa.gov";

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(192,168,0,177);

// Initialize the Ethernet client library
// with the IP address and port of the server
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
 pinMode(4, OUTPUT);
 digitalWrite(4, HIGH);

 // Open serial communications and wait for port to open:
 Serial.begin(9600);
  while (!Serial) {
   ; // wait for serial port to connect. Needed for Leonardo only
 }

 // start the Ethernet connection:
 if (Ethernet.begin(mac) == 0) {
   Serial.println("Failed to configure Ethernet using DHCP");
   // no point in carrying on, so do nothing forevermore:
   // try to congifure using IP address instead of DHCP:
   Ethernet.begin(mac, ip);
 }
 // give the Ethernet shield a second to initialize:
 delay(2000);
 Serial.println("connecting...");

 // if you get a connection, report back via serial:
 if (client.connect(server, 80)) {
   Serial.println("connected");
   // Make a HTTP request:
   client.println("GET /api/datagetter?date=latest&station=8516945&product=one_minute_water_level&datum=STND&units=metric&time_zone=gmt&application=web_services&format=xml HTTP/1.0");
   client.println("Host: tidesandcurrents.noaa.gov");
   client.println("Connection: close");
   client.println();
 }
 else {
   // kf you didn't get a connection to the server:
   Serial.println("connection failed");
 }
}

void loop()
{
 // if there are incoming bytes available
 // from the server, read them and print them:
 if (client.available()) {
   char c = client.read();
   Serial.print(c);
 }

// if the server's disconnected, stop the client:
 if (!client.connected()) {
   Serial.println();
   Serial.println("disconnecting.");
   client.stop();

   // do nothing forevermore:
   while(true);
 }
}


edit: Here is the serial monitor output.
Quote
connecting...
connected
HTTP/1.1 200 OK
Date: Wed, 20 Aug 2014 11:22:36 GMT
Server: GlassFish Server Open Source Edition 3.1.2.2
X-Powered-By: JSP/2.2
Cache-Control: no-cache,no-store,must-revalidate
Pragma: no-cache
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Content-Type: text/xml;charset=ISO-8859-1
Vary: User-Agent,Accept-Encoding
Connection: close
Set-Cookie: SERVERID=A; path=/

<?xml version="1.0" encoding="UTF-8" ?>
<data>
<metadata id="8516945"  name="Kings Point" lat="40.8103" lon="-73.7649"/>
<observations>
<ts t="2014-08-20 11:12"  v="6.020" />
</observations>
</data>

disconnecting.

bobbyzacharias

SurferTim, thanks so much! I now get the same output you showed. Next, we're on to parsing the XML to pull out the water height data, but there are some good examples out there of that so I'm confident we'll be able to do that.

Thanks again and I wonder if other people are having or will have DNS issues with the ethernet shield out there as well--if so, future forum readers, try referring to the server by IP rather than name and see if that solves your problem.

SurferTim

It will be only a few domains that will be affected. It appears to be when the domain name uses a CNAME (alias server name) for the main domain name.

SurferTim

The reason your client.connect() didn't work (returned connected when in fact it wasn't) is explained in this thread:
http://forum.arduino.cc/index.php?topic=262417.0

bobbyzacharias

Huh! I definitely agree with you and doughboy that this should be modified in future releases so poor suckers like doughboy and me don't waste half days on it any more.

bobbyzacharias

By the by, the project this question was part of ended up coming to a nice completion.

Please see rzach.me/tidetank for pictures, video, discussion, etc.

Thanks again for the help, PaulS and especially SurferTim!

Go Up