Ethernet Shield Complexity

Projects Networking, Protocols, and Devices
1

Hello: I follow a few sites with the arduino/ethernet shield combo.

I can observe planetary proximity (how close earth is to mars) on an astrology website (and probably many others): www.astrologyweekly.com

I'm also involved in winter travel periodically. So I check (among others):
www.wsdot.wa.gov for road conditions. At the road conditions site, you have to further refine your search down to specific mountain pass locations.

I can "pull" the html of astrologyweekly into my ethernet shield, but can't do that for the wsdot mountain passes sub-pages.

Is the HTML "Level" of the wsdot passes pages of an elevated release revision such that a little 8 bit microcontroller can't handle loading it????

I can't figure out why I can load the astrologyweekly page, but not the wsdot passes page????

If dhcp sort of moves this situation in the correct direction, I think I'll end up lost, because I've never dealt with it. I have code that I could submit, but it really is in need of a cleanup. That would take many days or weeks to accomplish.

Maybe someone could dig through this a bit, and steer me in the right direction????

2

Is the HTML "Level" of the wsdot passes pages of an elevated release revision such that a little 8 bit microcontroller can't handle loading it?

No.

If dhcp sort of moves this situation in the correct direction, I think I'll end up lost

Not to worry. A static IP address is as good as a dynamic IP address.

You make some kind of GET request to get the wsdot page. You get some kind of response. You showed neither one, so you really can't expect us to explain why what you get is not what you expect/sufficient.

3

The request to the wsdot site responds with a small amount of html. Early in that response there will be some verbeage: "html bad request" or something like that. surferTim posted some code a couple of years back, and I'm using variations of it to try to parse these pages. The code making the request to astrologyweekly is very nearly identical to the one going out to wsdot, but I get "bad request" on wsdot. ???

4

The code making the request

still hasn't been posted.

but I get "bad request" on wsdot.

My profile gives a clue where I live. I have an Arduino with ethernet shield. I could help...

5

This Code may move things in the right direction:

sprintf(outBuf,"GET %s HTTP/1.1",page);

I've lived all over the US. Am From way over in the East. Sort of live there now.

Is the HTML request from above, more or less, too primitive?

This system I'm running is with obviously the Arduino Compiler, and An Uno board with a Seeedstudio Ethernet shield. It's got me a bit stumped.

6

This Code may move things in the right direction:

sprintf(outBuf,"GET %s HTTP/1.1",page);

Well, now all we need to know is what is in page...

It's got me a bit stumped.

Too stumped to understand "POST YOUR CODE"?

7

I'm going to have to ask for some time to clean up the code. May not happen this PM. There's snippets of code in there from attempts to make things work, that didn't quite deliver. And alot of goofy varables.

8

monzt:
I'm going to have to ask for some time to clean up the code. May not happen this PM. There's snippets of code in there from attempts to make things work, that didn't quite deliver. And alot of goofy varables.

#7 below:

http://forum.arduino.cc/index.php/topic,148850.0.html

9

Hello:

Here is the code I have been using to "sift" through the HTML on www.astrologyweekly.com. I cleaned it up a little, but it still looks like a war zone:

/*
   Web client sketch for IDE v1.0.1 and w5100/w5200
   Uses GET method.
   Posted October 2012 by SurferTim
   Last modified September 15, 2013
*/

#include <SPI.h>
//#include <Ethernet.h>
#include <EthernetV2_0.h>
#include <SD.h>

File myFile;

// this must be unique
byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

// change to your network settings
IPAddress ip(192,168,2,2);
IPAddress gateway(192, 168, 2, 1);
IPAddress subnet(255, 255, 255, 0);

// change to your server
//IPAddress server(199,211,133,239);  // USNO
//IPAddress server(64,233,160,106); // Google
//IPAddress server(204,62,12,153); // timedotis
IPAddress server(192,254,198,32);  // www.astrologyweekly.com
//IPAddress server(204,72,144,10); // MSP Airport

//Change to your domain name for virtual servers
//char serverName[] = "tycho.usno.navy.mil/timer.html";
//char serverName[] = "www.google.com";
//char serverName[] = "time.is";
char serverName[] = "www.astrologyweekly.com";
//char serverName[] = "www.mspairport.com";
//char serverName[] = "www.mspairport.com/flight-Information/all-flights";
// If no domain name, use the ip address above
// char serverName[] = "74.125.227.16";

// change to your server's port
int serverPort = 80;

EthernetClient client;
int totalCount = 0;
char pageAdd[64];

// set this to the number of milliseconds delay
// this is 30 seconds
#define delayMillis 30000UL

unsigned long thisMillis = 0;
unsigned long lastMillis = 0;

void setup() {
  Serial.begin(9600);

  // disable SD SPI
//  pinMode(4,OUTPUT);
//  pinMode(10,OUTPUT);
//  digitalWrite(4,HIGH);

/**********************************

  if (!SD.begin(4))
  {
    Serial.println("SD initialization failed");
  }
**********************************/
//  myFile = SD.open("exblm.txt", FILE_WRITE);


  // Start ethernet
  Serial.println(F("Starting ethernet..."));
  //Ethernet.begin(mac, ip, gateway, gateway, subnet);

  // If using dhcp, comment out the line above 
  // and uncomment the next 2 lines plus the Ethernet.maintain call in loop

  if(!Ethernet.begin(mac)) Serial.println(F("failed"));
  else Serial.println(F("ok"));

  Serial.println(Ethernet.localIP());

  delay(2000);
  Serial.println(F("Ready"));
}

void loop()
{
  // If using dhcp to get an IP, uncomment the next line
   Ethernet.maintain();

  thisMillis = millis();

  if(thisMillis - lastMillis > delayMillis)
  {
    lastMillis = thisMillis;

    // Modify next line to load different page
    // or pass values to server
    sprintf(pageAdd,"/",totalCount);

    // sprintf(pageAdd,"/arduino.php?test=%u",totalCount);

    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Fail "));
    else Serial.print(F("Pass "));
    totalCount++;
    Serial.println(totalCount,DEC);
    for(;;)
      ;
  }    
}













byte getPage(IPAddress ipBuf,int thisPort, char *page)
{
  int inChar;
  int platT;
  int doneR;
  int q;
  int jumpuP;
  q = 0;
  platT = 0;
  doneR = 0;
  jumpuP = 0;
  char outBuf[128];
  char blubobaBuff[220];
  blubobaBuff[0] = 'P';
  blubobaBuff[1] = 'l';
  blubobaBuff[2] = 'a';
  blubobaBuff[3] = 'n';
  blubobaBuff[4] = 'e';
  blubobaBuff[5] = 't';
  blubobaBuff[6] = 's';
  blubobaBuff[7] = ' ';
  blubobaBuff[8] = 'N';
  blubobaBuff[9] = 'o';
  blubobaBuff[10] = 'w';
  blubobaBuff[11] = '<';
  blubobaBuff[12] = '/';
  blubobaBuff[13] = 'b';
  
  pinMode(4, OUTPUT);
  pinMode(10, OUTPUT);
  
  digitalWrite(4, HIGH);
  digitalWrite(10, HIGH);
  
  
  if (!SD.begin(4))
  {
    Serial.println("SD initialization failed");
  }
  

  myFile = SD.open("exblkk.txt", FILE_WRITE);

  

  Serial.print(F("connecting..."));

  if(client.connect(ipBuf,thisPort) == 1)
  {
    Serial.println(F("connected"));

    sprintf(outBuf,"GET %s HTTP/1.1",page);
    client.println(outBuf);
    sprintf(outBuf,"Host: %s",serverName);
    client.println(outBuf);
    client.println(F("Connection: close\r\n"));
  } 
  else
  {
    Serial.println(F("failed"));
    return 0;
  }

  // connectLoop controls the hardware fail timeout
  int connectLoop = 0;

  while(client.connected())
  {
    while(client.available())
    {
      inChar = client.read();
      if((!platT) && (!doneR))
      {
       if(((blubobaBuff[jumpuP]) != (inChar)) && (jumpuP < 11))
       {
         jumpuP = 0;
       }
       if(blubobaBuff[jumpuP] == inChar)
       {
         if(jumpuP >= 11)
         {
           platT = 1;
         }
         else
         {
           jumpuP++;
         }
       }
      }
      if((platT) && (!doneR))
      {
        blubobaBuff[jumpuP] = inChar;
        jumpuP++;
        
        
  /****  Altering next line up to like 94 gets more chacters
         in other words, increase jumpuP > 44 to: jumpUp > 94
  ****/

        if(jumpuP > 98)  // used to be jumpuP > 44
        {
          doneR = 1;
        }
      }

    }
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
 
    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)
    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }
  
  Serial.println();
  Serial.println();
  Serial.println();
 
 
  
  
  /****  Altering next line up to like 128 gets more chacters
         in other words, increase q < 48 to; q < 128
  ****/

  for(q = 12; q < 110; q++)  // used to be q < 48
  {
    Serial.print(blubobaBuff[q]);
    myFile.print(blubobaBuff[q]);
  }

  Serial.println();
 // myFile.println("Hello!");


  myFile.close();
  Serial.println(F("disconnecting."));
  // close client end
  client.stop();

  return 1;
}

By locating some of the info on the WSDOT page, I can pinpoint what HTML I want to be looking for. I'm not sure if the get command is not advanced enough to perform something. I can access astrology.com, but not on WSDOT. Maybe WSDOT doesn't want me in there?

Any help would be appreciated.

10 
// change to your server
//IPAddress server(199,211,133,239);  // USNO
//IPAddress server(64,233,160,106); // Google
//IPAddress server(204,62,12,153); // timedotis
IPAddress server(192,254,198,32);  // www.astrologyweekly.com
//IPAddress server(204,72,144,10); // MSP Airport

Is your delete key broken?

That mess with serverName is even worse.

#define delayMillis 30000UL

unsigned long thisMillis = 0;
unsigned long lastMillis = 0;

Using names that reflect the way people think makes more sense. Seeing delayTime, lastTime, thisTime, etc. causes me to say "OK, I know what that means...". That the value is in milliseconds is generally irrelevant.

//  pinMode(10,OUTPUT);

The SPI MOSI pin MUST be defines as OUTPUT. Commenting it out will simply cause the SPI devices (the ethernet shield and SD reader) stop working properly.

    sprintf(pageAdd,"/",totalCount);

What the hell is sprintf supposed to do with totalCount?

    if(!getPage(server,serverPort,pageAdd)) Serial.print(F("Fail "));

This IMPLIES that getPage() returns a boolean.

byte getPage(IPAddress ipBuf,int thisPort, char *page)

So, why doesn't it?

but it still looks like a war zone:

You better believe it.

You need to shitcan most of that mess. You need a function to make a GET request. You need a function to deal with the response from the server.

Neither function fucks around with the mode of any pins.

    sprintf(outBuf,"Host: %s",serverName);
    client.println(outBuf);

Far simpler to do:

    client.print(Host: ");
    client.print(serverName);

But, that is needed ONLY if the server is being hosted by a multi-domain-hosting company like godaddy. I can assure you that not a single US government page is hosted by godaddy.

      connectLoop = 0;
 
    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)
    {

The WTF light just exploded.

What does that convoluted mess have to do with getting data from wsdot?

11

So, I'll change the one line in the code:

   sprintf(pageAdd,"/",totalCount);

to something better, but there are other sprintf commands elsewhere, which I think would be best to keep using. I'm not too "up" on the details of sprintf. The only way to clean up most of this is to dig through documentation on C, and if I take the time to do that, this thread will go cold. It's alright with me if this thread doesn't go cold before I figure out how to get some of the other websites to "pull in". I don't know how to put a thread on "the back burner" and avoid having it go cold entirely.