Inconsistent HTTP Connnection

I am trying my hardest to get my Arduino Ethernet to communicate with our webserver at work. To start with I have been using the basic ethernet example that is already preloaded with the Arduino software.

Our webserver has a static IP address and I have been able to connect to it at times, but most times I can't. I'm feeling it has something to do with the server, as if I switch the IP address to a number of other sites all is fine.

Any ideas how to trouble shoot this?

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = {  0x90, 0xA2, 0xDA, 0x00, 0x95, 0xC9 };
IPAddress server(67,227,224,71); // 

// Initialize the Ethernet client library
// with the IP address and port of the server 
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
  // start the serial library:
  Serial.begin(9600);
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    for(;;)
      ;
  }
  // give the Ethernet shield a second to initialize:
  delay(1000);
  Serial.println("connecting...");

  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET / HTTP/1.0");
    client.println();
  } 
  else {
    // kf you didn't get a connection to the server:
    Serial.println("connection failed");
  }
}

void loop()
{
  // if there are incoming bytes available 
  // from the server, read them and print them:
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    // do nothing forevermore:
    for(;;)
      ;
  }
}

You are getting the "connection failed" message on the failed attempts? Have you tried disabling the microSD SPI? Any time these get weird, I check that first.

void setup()
{
   Serial.begin(9600);

   // disable the SD SPI interface
   pinMode(4,OUTPUT);
   digitalWrite(4,HIGH);

   // rest of your setup
}

edit: If you have access to the server logs, you might check there. That is how I found my problem. The SD SPI was corrupting the w5100 GET send to the server. The Apache log showed a malfomed request. It kinda looked like the GET I had sent, but all trashed up and stuff appended to it.

I checked the logs and it doesn't show anything. It shows that I was able to connect multiple times, which I was, but then nothing for the failed attempts.

If I create a loop and force it to keep retrying then it connects. This doesn't seem right, but I'm not sure what else to do.

So it works good for a while, then starts to get flakey, or in your case, goes straight to fail?
...and you did disable the SD SPI interface before testing it again?

edit: Just to make it clear, my connection never stayed trashy. I would see one or maybe two requests all trashed up after hundreds of good ones, then "connection failed" in the serial monitor, and not another entry in the server log.

I suspect you have two possible problem areas. The first is with using the DHCP functions. Your code appears to require resetting the arduino each time you want to run the code. This could cause issues with your router assigning a new ip address. The second is with your work server. Below is some client test code you can try. This code does not require a board reset to try the get request (just send an e from the serial monitor). First change the lan ip address from mine to a hard coded ip address for your lan and see if you can connect to my server. If that works, change the server ip address from my server to yours, to see if you can connect with your server. If you can repeatedly connect to your server, then the DHCP may have been causing an issue. if you can't connect you may have an issue with your server.

//zoomkat 12-08-11
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605 

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////

void setup(){

  Ethernet.begin(mac, ip);
  //Ethernet.begin(mac, ip, subnet, gateway);
  Serial.begin(9600); 
  Serial.println("Better client test 12/01/11"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(myserver, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

@zoomkat: You should remove this from your code. People might think it is correct, and it isn't for IDE v1.0.

  //Ethernet.begin(mac, ip, subnet, gateway);

Here are the new v1.0 formats.
Ethernet.begin(mac);
Ethernet.begin(mac,ip);
Ethernet.begin(mac,ip,dns);
Ethernet.begin(mac,ip,dns,gateway);
Ethernet.begin(mac,ip,dns,gateway,subnet);

If you are not going to use dns, use the gateway for the dns server, not the subnet. Like this:
Ethernet.begin(mac,ip,gateway,gateway,subnet);

Sorry for the delayed response.

SurferTim: It's intermittent, but more often it doesn't work. I haven't disabled the SD SPI. How do I do that?

zoomkat: I tried your code and things again seem intermittent. It connects right away, other times it fails after 10-20 seconds or so. Some times when it does connect, it seems to hang halfway through downloading the content.

Accept-Ranges: bytes
Content-Length: 51
Connection: close
Content-Type: text/plain; charset=UT

I'm now getting mixed results with some of my code that uses my web page server for testing. Below is some code modified by another to connect to a time server. You might check to see if it will connect more consistantly.

//zoomkat 12-08-11, combined client and server
//simple button GET with iframe code
//for use with IDE 1.0
//open serial monitor and send an g to test client and
//see what the arduino client/server receives
//web page buttons make pin 4 high/low
//use the \ slash to escape the " in the html 
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields


//NTP server implementation added by RobertJP

#include <SPI.h>
#include <Ethernet.h>
#include <EthernetUdp.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
IPAddress ip(192,168,1,102); // ip in lan
IPAddress gateway(192,168,1,1); // internet access via router
IPAddress subnet(255,255,255,0); //subnet mask
IPAddress myserver(208,104,2,86); // zoomkat web page
EthernetServer server(84); //server port
EthernetClient client;
String readString; 

//////////////////////NTP declarations
IPAddress timeServer(77,245,91,218); // time.nist.gov NTP server
const int NTP_PACKET_SIZE= 48; // NTP time stamp is in the first 48 bytes of the message
byte packetBuffer[ NTP_PACKET_SIZE]; //buffer to hold incoming and outgoing packets 
EthernetUDP Udp; // A UDP instance to let us send and receive packets over UDP
unsigned int localPort = 8888; //Port to listen for UDP packets

void setup(){
  pinMode(5, OUTPUT); //pin selected to control
  Ethernet.begin(mac,ip,gateway,gateway,subnet);
  server.begin();
  Serial.begin(9600); 
  Serial.println("server/client 1.0 test 12/08/11"); // keep track of what is loaded
  Serial.println("Send an g in serial monitor to test client"); // what to do to test client
  Udp.begin(localPort);
  Serial.println("UDP started");
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) 
  {
    byte inChar;
    inChar = Serial.read();
    if(inChar == 'g')
    {
      sendGET(); // call sendGET function
    }
  }  

  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {

          //store characters to string 
          readString += c; 
          //Serial.print(c);
        } 

        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.println(readString); //print to serial monitor for debuging 

            //now output HTML data header
          if(readString.indexOf('?') >=0) { //don't send new page
            client.println("HTTP/1.1 204 Robert");
            client.println();
            client.println();  
          }
          else {
            client.println("HTTP/1.1 200 OK"); //send new page
            client.println("Content-Type: text/html");
            client.println();

            client.println("<HTML>");
            client.println("<HEAD>");
            client.println("<TITLE>Arduino GET test page</TITLE>");
            client.println("</HEAD>");
            client.println("<BODY>");

            client.println("<H1>Zoomkat's simple Arduino 1.0 button</H1>");

            client.println("<a href=\"/?on\" target=\"inlineframe\">ON</a>"); 
            client.println("<a href=\"/?off\" target=\"inlineframe\">OFF</a>"); 

            //client.println("<IFRAME name=inlineframe src=\"res://D:/WINDOWS/dnserror.htm\" width=1 height=1\">");
            client.println("<IFRAME name=inlineframe style=\"display:none\" >");          
            client.println("</IFRAME>");

            client.println("</BODY>");
            client.println("</HTML>");
          }

          delay(1);
          //stopping client
          client.stop();

          ///////////////////// control arduino pin
          if(readString.indexOf("on") >0)//checks for on
          {
            digitalWrite(5, HIGH);    // set pin 4 high
            Serial.println("Led On");
          }
          if(readString.indexOf("off") >0)//checks for off
          {
            digitalWrite(5, LOW);    // set pin 4 low
            Serial.println("Led Off");
          }
          //clearing string for next read
          readString="";

        }
      }
    }
  }
} 

void sendGET()
{
  Serial.println("Send GET");
  sendNTPpacket(timeServer); // send an NTP packet to a time server
   // wait to see if a reply is available
  delay(1000);  
  if ( Udp.parsePacket() ) { 
    Serial.println("Got it!"); 
    // We've received a packet, read the data from it
    Udp.read(packetBuffer,NTP_PACKET_SIZE);  // read the packet into the buffer

    //the timestamp starts at byte 40 of the received packet and is four bytes,
    // or two words, long. First, esxtract the two words:

    unsigned long highWord = word(packetBuffer[40], packetBuffer[41]);
    unsigned long lowWord = word(packetBuffer[42], packetBuffer[43]);  
    // combine the four bytes (two words) into a long integer
    // this is NTP time (seconds since Jan 1 1900):
    unsigned long secsSince1900 = highWord << 16 | lowWord;  
    Serial.print("Seconds since Jan 1 1900 = " );
    Serial.println(secsSince1900);               

    // now convert NTP time into everyday time:
    Serial.print("Unix time = ");
    // Unix time starts on Jan 1 1970. In seconds, that's 2208988800:
    const unsigned long seventyYears = 2208988800UL;     
    // subtract seventy years:
    unsigned long epoch = secsSince1900 - seventyYears;  
    // print Unix time:
    Serial.println(epoch);                               


    // print the hour, minute and second:
    Serial.print("The UTC time is ");       // UTC is the time at Greenwich Meridian (GMT)
    Serial.print(1+(epoch  % 86400L) / 3600); // print the hour (86400 equals secs per day)
    Serial.print(':');  
    if ( ((epoch % 3600) / 60) < 10 ) {
      // In the first 10 minutes of each hour, we'll want a leading '0'
      Serial.print('0');
    }
    Serial.print((epoch  % 3600) / 60); // print the minute (3600 equals secs per minute)
    Serial.print(':'); 
    if ( (epoch % 60) < 10 ) {
      // In the first 10 seconds of each minute, we'll want a leading '0'
      Serial.print('0');
    }
    Serial.println(epoch %60); // print the second
  }
  Serial.println("Done!");
}

// send an NTP request to the time server at the given address 
unsigned long sendNTPpacket(IPAddress& address)
{
  // set all bytes in the buffer to 0
  memset(packetBuffer, 0, NTP_PACKET_SIZE); 
  // Initialize values needed to form NTP request
  // (see URL above for details on the packets)
  packetBuffer[0] = 0b11100011;   // LI, Version, Mode
  packetBuffer[1] = 0;     // Stratum, or type of clock
  packetBuffer[2] = 6;     // Polling Interval
  packetBuffer[3] = 0xEC;  // Peer Clock Precision
  // 8 bytes of zero for Root Delay & Root Dispersion
  packetBuffer[12]  = 49; 
  packetBuffer[13]  = 0x4E;
  packetBuffer[14]  = 49;
  packetBuffer[15]  = 52;

  // all NTP fields have been given values, now
  // you can send a packet requesting a timestamp:         
  Udp.beginPacket(address, 123); //NTP requests are to port 123
  Udp.write(packetBuffer,NTP_PACKET_SIZE);
  Udp.endPacket(); 
}

It gets the time fine, but not a webpage. Haven't read through the code deep enough to know exactly what it is suppose to be returning...

It gets the time fine, but not a webpage.

You may need to modify the arduino server ip address and gateway ip address to be in alignment with your router's ip address. The router ip address (gateway) is often { 192, 168, 1, 1 } or { 192, 168, 0, 1 } . The arduino server ip address needs to match the first three numbers in the router ip address.

If I did not disable the SD SPI, my NTP time would wander due to trash on the SPI bus. It would be fine for a few updates, then start to wander.

void setup()
{
   Serial.begin(9600);

   // this disables the SD SPI interface.
   pinMode(4,OUTPUT);
   digitalWrite(4,HIGH);

   // rest of the setup
}

If the NTP function is working, your network settings should be ok.

The NTP function (or any UDP service) was affected most by the SD SPI corruption, so that may not be your problem. I do have a 2GB microSD card in the slot, and I think that may increase the odds of corruption due to the SD SPI interface.

What about the web page is not working? You know that code has the server set on port 84, not 80, correct? If you use the ip in zoomkat's code, this is the URL to access it from your localnet.
http://192.168.1.102:84

edit: or you can change the server port to port 80 so it will respond with a "normal" URL.

EthernetServer server(80); //server port

I do have a 2GB microSD card in the slot, and I think that may increase the odds of corruption due to the SD SPI interface.

Having the card inserted provides an electrical connection to pin 4 that can cause issues, depending on the circutitry in the card and how the card is operating when inserted and energized. Have you tried your code with issues with the card removed? Disableing pin 4 should not be required if nothing is connected to pin 4.

SurferTim: So after a little fiddling I got the code working and was able to access the webpage fine. But things seem to work fine on the local side of things. It's when I try to reach out and grab pages that things get wonky (i.e. failing to connect multiple times, only half the page downloading, etc.)

zoomkat: I don't believe I even have a card installed at the moment. I'll double check that when I get home. As for the server and gateway IP addresses, I've double checked them and things seem fine. My network uses 10.0.1.*, would this cause issues?

Can you explain "reach out and grab pages"? I thought this was server code. Clients "grab pages".

My network uses 10.0.1.*, would this cause issues?

That is a localnet ip range. It will be good as a server ip only within the localnet or the local router. From the internet, you would need to translate your public ip to the internal localnet ip.

Uh oh, looks like we got crossed up. I"m trying to set the Arduino up as a client, not a server.

That was my bad.

Which code are you using now? Are you still trying your original code? Or zoomkat's?

I haven't had a chance to work on this project for the past few days, but I tried using my original code (with a few modifications) as well as zoomkat's. Both seem to give me mixed results as far as connectivity. With zoomkat's, sometimes it connects and downloads a page immediately after I press 'e', other times it takes around 20 seconds before it says connection failed.

I originally had things set up in a for loop so that it would keep trying over and over again. This seemed to provide okay results, but also seemed like a work-around with addressing the actual problem.

This loads www.google.com every 10 seconds. Don't do that to Google too long. Besides, it is a big page and takes a while to display at 9600 baud. Change the server ip to yours.

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
IPAddress ip(192,168,2,2);
IPAddress gateway(192, 168, 2, 1);
IPAddress subnet(255, 255, 255, 0);

IPAddress server(74,125,227,16); // Google

EthernetClient client;
int totalCount = 0;
int loopCount = 0;

void setup() {
  Serial.begin(9600);

  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  delay(2000);
  Serial.println("Ready");
}

void loop()
{
  if(loopCount < 10)
  {
    delay(1000);
  }
  else
  {
    loopCount = 0;

    if(!getPage(server,"/")) Serial.print("Fail ");
    else Serial.print("Pass ");
    totalCount++;
    Serial.println(totalCount,DEC);
  }    

  loopCount++;
}

byte getPage(IPAddress ipBuf,char *page)
{
  int inChar;
  char outBuf[64];

  Serial.print("connecting...");

  if(client.connect(ipBuf,80))
  {
    Serial.println("connected");

    sprintf(outBuf,"GET %s HTTP/1.0\r\n\r\n",page);
    client.write(outBuf);
  } 
  else
  {
    Serial.println("failed");
    return 0;
  }

  while(client.connected())
  {
    while(client.available())
    {
      inChar = client.read();
      Serial.write(inChar);
    }
  }

  Serial.println();

  Serial.println("disconnecting.");
  client.stop();

  return 1;
}

Hmmm, well that seems to work so far. I've only had a few minutes this morning to play, but I haven't gotten any failed attempts (except for when I forgot to change the IP).

So what's different? Why is this one connecting fine whereas the others weren't?

jamgood96:
Hmmm, well that seems to work so far. I've only had a few minutes this morning to play, but I haven't gotten any failed attempts (except for when I forgot to change the IP).

So what's different? Why is this one connecting fine whereas the others weren't?

The others were not mine. :slight_smile:

edit: There are some reasons mine works better. I know how tcp works. If you look at the getPage() function, you will notice the "while(client.connected()) while(client.available())" loops.

If you are downloading a big page (like Google's home page), you will get several packets. That loop will check client.available(), and for a while, it will be zero. Then it will be about 1400. Your goal is to empty that rx buffer as fast as you can. Get client.available() to zero. Then it may be zero again for a few checks, then it will be 1400 again (another packet).

The client.connected() will return true until you have emptied the rx buffer of the final packet, so as long as client.available() returns non-zero, you are connected.