Ethernet shield fails to connect to weather API, Telnet works, Google works

Hey All,

I've been struggling with this one for a while now and I could really use some help.

I have a simple sketch that is supposed to return the temperature outside and put it on an lcd.

The ethernet shield works with the httpclient example and has worked with both the yahoo api and the Weather Underground API.

The problem I have is that it won't work anymore. Take an identical piece of code, swap in a different server (e.g. www.google.com) and it connects and you get a response, but no connection from the two api's.

The API's are both up and running and the information is right because I can telnet to them just fine.

Is it possible that they have both blocked the mac address of the arduino somehow? Any other ideas? Thanks in advance.

Joe

/*
  Web client
 
 This sketch connects to a website (http://www.google.com)
 using an Arduino Wiznet Ethernet shield. 
 
 Circuit:
 * Ethernet shield attached to pins 10, 11, 12, 13
 
 created 18 Dec 2009
 by David A. Mellis
 modified 9 Apr 2012
 by Tom Igoe, based on work by Adrian McEwen
 
 */

#include <SPI.h>
#include <Ethernet.h>
#include <LiquidCrystal.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = {0x90,0xA2,0xDA,0x0D,0x4A,0x03};
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(74,125,232,128);  // numeric IP for Google (no DNS)
char server[] = "www.google.com";    // name address for Google (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(10,0,1,100);

LiquidCrystal lcd(3, 4, 5, 6, 7, 8);

// Initialize the Ethernet client library
// with the IP address and port of the server 
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
 // Open serial communications and wait for port to open:
  Serial.begin(9600);
    lcd.begin(16, 2);
   while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }

  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    // try to congifure using IP address instead of DHCP:
    Ethernet.begin(mac, ip);
  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  Serial.println("connecting...");
  lcd.setCursor( 0, 0); 
  lcd.print("connecting");
  lcd.setCursor(0, 1);
  lcd.print(Ethernet.localIP());

  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected");
      lcd.setCursor( 0, 0); 
  lcd.print("connected");
    // Make a HTTP request:
    client.println("GET /search?q=arduino HTTP/1.1");
    client.println("Host: www.google.com");
    client.println("Connection: close");
    client.println();
  } 
  else {
    // kf you didn't get a connection to the server:
    Serial.println("connection failed");
  }
}

void loop()
{
  // if there are incoming bytes available 
  // from the server, read them and print them:
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
    lcd.print(c);
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    // do nothing forevermore:
    while(true);
  }
}

You can test that theory by changing your mac address. Like this:

byte mac[] = {0x90,0xA2,0xDA,0x0D,0x40,0x73};

Thanks! That worked. The request went through ok.

Two questions follow through:

1 - How can it work with the wrong MAC address?
2 - What did I do to piss them off? I don't want to do it again. I originally did not have the "Connection: close" line in my GET request. Could that be what blacklisted me?

Joe

The second attempt failed. Am I blocked again already? This is nuts.

crambam:
The second attempt failed. Am I blocked again already? This is nuts.

I wouldn't think so. What URL are you trying? What page? I will try it with my client code.

Thanks again for the help. I've provided the whole sketch which is supposed to display the temp on an lcd and contrast with internal temperature. I've also provided the relevant snippets at the top. server and connect statement.

I'm truly puzzled here, after changing the mac address it works once, then fails the next time.

char server[] = "weather.yahooapis.com";
EthernetClient client;
client.connect(server, 80);
#include <SPI.h>
#include <Ethernet.h>
#include <LiquidCrystal.h>


int tempPin =0;
LiquidCrystal lcd(3, 4, 5, 6, 7, 8);
byte mac[] = {0x90,0xA2,0xDA,0x0D,0x4A,0x03};
char server[] = "weather.yahooapis.com";
byte ipAddress[] = {38,102,136,138};
float temp = 999;

IPAddress ip(10,0,1,100);

EthernetClient client;

void setup()
{
  Serial.begin(9600);
  lcd.begin(16, 2);
  lcd.print("booting up");
  
  if (Ethernet.begin(mac) ==0)
  {
    lcd.setCursor( 0, 0); 
    lcd.print("DHCP Failed"); 
    Ethernet.begin(mac, ip);
  }
  
  delay(1000);
  Serial.println("connecting...");
  lcd.setCursor( 0, 0); 
  lcd.print("connecting");
  lcd.setCursor(0, 1);
  lcd.print(Ethernet.localIP());
  
  if (client.connect(server, 80))
  {
    lcd.setCursor( 0, 0);
    lcd.print("connected  ");
    Serial.println("connected  ");
    delay(2000);
    Serial.println("GET /forecastrss?w=2502265&u=c HTTP/1.1");
    Serial.println("Host: weather.yahooapis.com");
    Serial.println();
    client.println("GET /forecastrss?w=2502265&u=c HTTP/1.1");
    client.println("Host: weather.yahooapis.com");
    client.println("Connection: close");
    client.println();
  }
  else
  {
    lcd.setCursor( 0, 0);
    lcd.print("connection failed");
    Serial.println("connection failed");
  }

 
}

void loop()
{
  if (client.available()) {
      if (temp==999){ // no temp retrieved yet
        char c = client.read();
        if (c =='y'){
          c = client.read();
           if (c =='w'){
            c = client.read();
              if (c =='e'){
                 c = client.read();
                  if (c =='a'){
                    c = client.read();
                      if (c =='t'){
                        c = client.read();
                          if (c =='h'){
                           c = client.read();                           
                            if (c =='e'){
                              c = client.read();
                                    if (c =='r'){
                                      c = client.read();
                                        if (c ==':'){
                                          c = client.read();
                                            if (c =='c'){
                                              c = client.read();
                                                if (c =='o'){
                                                 c = client.read();
                                                  if (c =='n'){
                                                    c = client.read();
                                                      if (c =='d'){
                                                        c = client.read();
                                                          if (c =='i'){
                                                           c = client.read();
                                                            if (c =='t'){
                                                             c = client.read();
                                                              if (c =='i'){
                                                                c = client.read();
                                                                  if (c =='o'){
                                                                    c = client.read();
                                                                      if (c =='n'){
                                                                        for (int i=0;i<40;i++)
                                                                        c=client.read(); 
                                                                        float temp = client.parseFloat();
                                                                        lcd.print("Outside temp     C  ");
                                                                        lcd.setCursor(6,0);
                                                                        lcd.print(temp);
      }}}}}}}}}}}}}}}}}}}
                                                            
      lcd.setCursor(0,1);
      int tempReading = analogRead(tempPin);
      float tempVolts = tempReading * 5.0 /1024.0;
      float tempC = (tempVolts - 0.5) * 100.0;
      lcd.print("Inside Temp     C ");
      lcd.setCursor(13,1);
      lcd.print(tempC);
      delay(500);
    }

  if (!client.connected())
  {
    lcd.setCursor(0,0);
    lcd.print("disconnecting.");
    client.stop();
    
    while(true);
  }
}

I should add that it worked several times yesterday. I was able to access the yahoo api at least 10 times, but then nothing.

This also happened with wunderground.com it worked a couple of days ago, and now nothing.

as another test, I tried manually setting my internal IP instead of DHCP. This worked, but my guess is that this will only work for a couple of tries (2 successful connections so far). Maybe its a router or isp thing?

This is working right now on mine. I modified it somewhat to suit your requirements. The site says that is not a valid city. I'm on pass 9 now. It takes 30 seconds to start.

/*
   Web client sketch for IDE v1.0.1 and w5100/w5200
   Uses GET method.
   Posted October 2012 by SurferTim
   Last modified June 17, 2013
*/

#include <SPI.h>
#include <Ethernet.h>

// this must be unique
byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

char server[] = "weather.yahooapis.com";

int serverPort = 80;

EthernetClient client;
int totalCount = 0;
int loopCount = 0;
char pageAdd[32];

void setup() {
  Serial.begin(9600);

  // disable SD SPI
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  // Start ethernet
  Serial.println(F("Starting ethernet..."));
  if(!Ethernet.begin(mac)) Serial.println(F("failed"));
  else Serial.println(F("ok"));
  digitalWrite(10,HIGH);

  Serial.println(Ethernet.localIP());

  delay(2000);
  Serial.println(F("Ready"));
}

void loop()
{
  if(loopCount < 30)
  {
    // if loopCount is less than 30, just delay a second
    delay(1000);
  }
  else
  {
    // every thirty seconds this runs

    // check ip lease renewal
    int ethStatus = Ethernet.maintain();

    if(ethStatus) {
        Serial.print(F("DHCP check..."));

        switch(ethStatus) {
            case 1: Serial.println(F("renew fail"));
                       break;
            case 2: Serial.println(F("renew ok"));
                       break;
            case 3: Serial.println(F("rebind fail"));
                       break;
            case 4: Serial.println(F("rebind ok"));
                       break;
        }
    }

    loopCount = 0;

    // Modify next line to load different page
    // or pass values to server
    sprintf(pageAdd,"/forecastrss?w=2502265&u=c");

    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Fail "));
    else Serial.print(F("Pass "));
    totalCount++;
    Serial.println(totalCount,DEC);
  }    

  loopCount++;
}

byte getPage(char *ipBuf,int thisPort, char *page)
{
  int inChar;
  char outBuf[128];

  Serial.print(F("connecting..."));

  if(client.connect(ipBuf,thisPort))
  {
    Serial.println(F("connected"));

    sprintf(outBuf,"GET %s HTTP/1.1",page);
    client.println(outBuf);
    sprintf(outBuf,"Host: %s",ipBuf);
    client.println(outBuf);
    client.println(F("Connection: close\r\n"));
  } 
  else
  {
    Serial.println(F("failed"));
    return 0;
  }

  // connectLoop controls the hardware fail timeout
  int connectLoop = 0;

  while(client.connected())
  {
    while(client.available())
    {
      inChar = client.read();
      Serial.write(inChar);
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
    }

    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)
    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }

  Serial.println();

  Serial.println(F("disconnecting."));
  // close client end
  client.stop();

  return 1;
}

edit: It works ok if you change the page to this. The @ doesn't work there.
/forecastrss?w=2502265&u=c

I wonder if my problem comes from the frequent refreshes? your code moves the api query into the loop which means you can leave the arduino connected with the same IP. Is it possible my local network is just denying access to a device that is getting a new IP address every time I push a new build? I'm going to try to amen my code to reflect yours, (move the api call inside the loop). Then try leaving it active for a little while.

Any Idea what a reasonable update frequency is for api calls? 30 seconds seems very frequent.

So far it is working ok here. I made the change to the page, and I presume you are getting the weather for Sunnyvale, Ca.

Change it to whatever time period you want. I had 30 seconds set in the playground.

The router doesn't know about the build. If you don't change the mac address, you should get the same ip for a while. I'm using dhcp.

There is one thing about dhcp. You must call Ethernet.maintain() in the loop() to renew the ip lease.

Thanks for all the help Tim. It's great to see that your code is working properly here. I'm going to try to refactor what I've got to mimic yours a bit more. I'll update once I have some progress.

Thanks!

If you use dhcp, don't forget to put the Ethernet.maintain() call in the loop(). You wouldn't want your ip to expire while the Arduino is using it. I use static ips.

edit: I added the Ethenet.maintain() call to the code above. Every minute or so should be plenty. The call doesn't do anything until the ip lease has reached half the expiration time.

I'm doing the same that you did in weather.yahooapis.com but using a SparkFun WiFly and it does not work at all. When doing the same with weather underground, it works like a charm but WU is not an RSS feed.
I'm puzzled because it only works when the host is added in the GET statement, but then it only retrieves 1460 chars and cuts the connection. :astonished:
Ever used the WiFly? Any help would be greatly appreciated.