Ethernet HTTP 1.1 - again

Projects Networking, Protocols, and Devices
1

A lot of words said about Ethernet shield and it's library. Have anyone got stable version fo mere http continuos GET requests with Arduino Mega 2560 + Ethernet shield with WS5100 ?
Mine works fine on some sketches for the several first requests but finally remote web-server refuse to deal with arduino ((
I heard some people faced the same story and it was said that problem might be bad implementation of "client.stop" function and/or incorrect handling of keep-alive procedure by Ethernet shield. I checked the matter of overheating - the shield is a bit hot, but it sends data perfectly to local server in home network without any problems. So , the problem appears than i have to add "HTTP/ 1.1" and all that stuff related to HTTP transmission over real Internet.
Here's a routine that works only some time. After some successfull requests arduino still successfully connects, but getdata.php on server side do not receive any data. I've tried to approach than getdata.php via browser - it still works fine, so the problem is not in that script.

String msg = "GET /getdata.php?i="+String(i)+ "&t="+String(wv)+ " HTTP/1.1";

#define TEST_MODE 1
if(TEST_MODE)
{
	Serial.println(Ethernet.localIP()); 
	Serial.println(msg);
}
if (client.connect(server,80))   {if(TEST_MODE) Serial.println("ok, connected!");}
else                             {if(TEST_MODE) Serial.println("failed to connect...");}

client.println(msg);    
client.println("Host: www.my-website.com");
client.println();
 // if (client.available()) {char c = client.read(); Serial.println(c); }
 //client.println("Connection: close");
delay (5000);
client.stop();
delay (30000);

please disregard name of site, it was changed. sorry for pure English, not my native one.

2

This works for me. It has all the stuff to keep it from failing.
http://playground.arduino.cc/Code/WebClient

3

Thanks, SurferTim.

That's pretty fresh example (updated 23 aug 2013), and it works perfect for me continuously (on google). While than i try to listen my vitrual host (by uncommenting "client.read" above, i receive soon some stuff "bla bla bla... this site is temporary unavailable" :smiley: instead of "OK" from my php script. Meantime than i try to run script via browser, it still works fine. So, it seems question goes to my hoster. I guess some hoster's routine disables access to the site than it get continuously "strange" and short http header which arduino generates. Any other ideas?

May b i have to try full header to emulate normal browser, like here https://sites.google.com/site/arduinonetserver/webserver/ib_request but for some reason i think it will not help..

4

This is not correct. The "Connection: close" must be in the header.

client.println(msg);    
client.println("Host: www.my-website.com");
client.println();
 // if (client.available()) {char c = client.read(); Serial.println(c); }
 //client.println("Connection: close");

Without the timeout code, this should work.

client.println(msg);    
client.println("Host: www.my-website.com");
client.println("Connection: close\r\n");

while(client.connected()) {
 while (client.available()) {char c = client.read(); Serial.println(c); }
}
client.stop();

If the connection breaks, it will lock up in the "while(client.connected())" loop without the timeout code.

5

Ohhhh Lord!)))

Thank you, kind man... That while cyrcle makes me so happy :smiley:
There are still some lost requests from time to time but generally it's much better than before. May b i just need to make a bit bigger delay.

6

Some server test code that seems fairly stable for repeated request. Has a counter that displays the number of updates.

// zoomkat's meta refresh data frame test page 5/25/13
// use http://192.168.1.102:84 in your brouser for main page
// http://192.168.1.102:84/data static data page
// http://192.168.1.102:84/datastart meta refresh data page
// for use with W5100 based ethernet shields
// set the refresh rate to 0 for fastest update
// use STOP for single data updates

#include <SPI.h>
#include <Ethernet.h>

const int analogInPin0 = A0;
const int analogInPin1 = A1;
const int analogInPin2 = A2;
const int analogInPin3 = A3;
const int analogInPin4 = A4;
const int analogInPin5 = A5;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // arduino ip in lan
byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port
unsigned long int x=0; //set refresh counter to 0
String readString; 

//////////////////////

void setup(){
  Serial.begin(9600);
    // disable SD SPI if memory card in the uSD slot
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  Ethernet.begin(mac, ip, gateway, gateway, subnet);
  server.begin();
  Serial.println("meta refresh data frame test 5/25/13"); // so I can keep track of what is loaded
}

void loop(){
  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();
         if (readString.length() < 100) {
          readString += c; 
         } 
        //check if HTTP request has ended
        if (c == '\n') {

          //check get atring received
          Serial.println(readString);

          //output HTML data header
          client.println("HTTP/1.1 200 OK");
          client.println("Content-Type: text/html");
          client.println();

          //generate data page
          if(readString.indexOf("data") >0) {  //checks for "data" page
            x=x+1; //page upload counter
            client.print("<HTML><HEAD>");
            //meta-refresh page every 1 seconds if "datastart" page
            if(readString.indexOf("datastart") >0) client.print("<meta http-equiv='refresh' content='1'>"); 
            //meta-refresh 0 for fast data
            if(readString.indexOf("datafast") >0) client.print("<meta http-equiv='refresh' content='0'>"); 
            client.print("<title>Zoomkat's meta-refresh test</title></head><BODY>
");
            client.print("page refresh number: ");
            client.print(x); //current refresh count
            client.print("

");
            
              //output the value of each analog input pin
            client.print("analog input0 is: ");
            client.print(analogRead(analogInPin0));
            
            client.print("
analog input1 is: ");
            client.print(analogRead(analogInPin1));
                        
            client.print("
analog input2 is: ");
            client.print(analogRead(analogInPin2));
            
            client.print("
analog input3 is: ");
            client.print(analogRead(analogInPin3));
                                    
            client.print("
analog input4 is: ");
            client.print(analogRead(analogInPin4));
            
            client.print("
analog input5 is: ");
            client.print(analogRead(analogInPin5));
            client.println("
</BODY></HTML>");
           }
          //generate main page with iframe
          else
          {
            client.print("<HTML><HEAD><TITLE>Zoomkat's frame refresh test</TITLE></HEAD>");
            client.print("Zoomkat's Arduino frame meta refresh test 5/25/13");
            client.print("

Arduino analog input data frame:
");
            client.print("&nbsp;&nbsp;<a href='http://192.168.1.102:84/datastart' target='DataBox' title=''yy''>META-REFRESH</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/data' target='DataBox' title=''xx''>SINGLE-STOP</a>");
            client.print("&nbsp;&nbsp;&nbsp;&nbsp;<a href='http://192.168.1.102:84/datafast' target='DataBox' title=''zz''>FAST-DATA</a>
");
            client.print("<iframe src='http://192.168.1.102:84/data' width='350' height='250' name='DataBox'>");
            client.print("</iframe>
</HTML>");
          }
          delay(1);
          //stopping client
          client.stop();
          //clearing string for next read
          readString="";
        }
      }
    }
  }
}
7

The suggestion to those who suffer problems with this shield - note, it seems to be VERY sensitive to the quality of your line.
If you have problems like "sometimes connects, sometimes fails" and you are using fresh version of Ethernet lib, you can try the following.
Connect your shield to the internet not directly but via some computer, for example via notebook with two network cards (one Wi-Fi and another one - ethernet).
To do this for example on Win7 you are to configure your wi-fi connection to allow sharing of internet access from your ethenet card connections, i.e. in properties of your wi-fi connection choose "access" -> "allow other users to connect via this connection" and than choose your ehternet connection. You are to assign ip address on 192.168.37.1 to your ehternet card (it's for some reason standard suggestion of Win7). So, your arduino is also to be configured for own address something like 192.168.137.2, and server 192.168.137.1.
Sure, you will be surprised with stability of this connection.

Would also suggest

  • define some veriable for test mode and send all the data to seri
  • try several attempts of sending data for each connection if first attempt failed. On my bad line with below listed code i managed to send request on 2...6 attempt. It will try to send your message ten times, and it really often helps.
    It helps for more stable connection but anyway not guarantee from "failed to connect" finally. The SAME code via notebooks' connection (as listed above) gave stable results during several days. No any single byte lost !
    Meantime if i connect my home pc (or notebook) to that ethernet "bad" line, that connection is also quite stable. So, seems that W5100 based ethernet shield's hardware (or library) is less reliable that ordinary ethernet pc card. On local connections ethernet shield- PC (with denver server) arduino shield is alsow works perfectly nice. That means that problem is most likely with timings and most of standard internet connections will be "bad" for the shield, while local ones will be ok.
#define TEST_MODE 1
Ethernet.localIP();  
int attempts = 0;
int BytesSent = 0;
int your_data = 77777; // here is data you wanna pass to server
// data.php - is name of php-file on your server that handles your data

String msg = "GET /data.php?i="+String(your_data)+ " HTTP/1.1";

if (client.connect(server,80))  
     {  if(TEST_MODE) Serial.println("ok, connected! ");
		do { BytesSent = client.println(msg); 
                     client.println("Host: www.your-host-name-here.com");
                     client.println("Connection: close\r\n");   
                     client.println();
                     while(client.connected()) 
	          {  while (client.available())     {char c = client.read(); if (TEST_MODE) Serial.print(c); }}
                     
                     attempts++; 
                     if (TEST_MODE) {Serial.print(BytesSent); Serial.print(" attempt # "); Serial.println(attempts); }
                     client.stop(); delay (5000); client.connect(server,80);
                    }           
                while (!BytesSent && attempts < 9); 		
 
                if (TEST_MODE) 
                    {  if (!BytesSent)  Serial.println("failed to send message...");
                       else Serial.println("Message sent. Now some delay..."); Serial.println();  
                    }
                delay (100);
		
     }
else  if (TEST_MODE) {Serial.println("Uuupsss... Failed to connect server...");}

while(client.connected()) 
	 {  while (client.available()) {char c = client.read(); if (TEST_MODE) Serial.print(c); }}
delay (500);
client.stop();
delay (65000); // delay until next message ready to be sent
}
8

A lot of words said about Ethernet shield and it's library. Have anyone got stable version fo mere http continuos GET requests with Arduino Mega 2560 + Ethernet shield with WS5100 ?

Here was your answer:

SurferTim:
This works for me. It has all the stuff to keep it from failing.
Arduino Playground - HomePage

BTW, you are still short the response timeout feature. If you want bulletproof, it needs that, or you will be back here in a few days with "it fails every few days, and I must reset it to get it to work again".

edit: I forgot to show you where your code will lock up every day or so.

while(client.connected()) 
	 {  while (client.available()) {char c = client.read(); if (TEST_MODE) Serial.print(c); }}

If the connection breaks (fails) or the server stalls, it will never exit the "while(client.connected())" loop.

9

Thanks again. I'm just starting with arduino, so a couple of simple questions regarding that example on your link.
I did'nt catch well this code, but is where any specific reason for using sprintf instead of client.print ?
How should i modify GetPage funciton if i have only server name and no ip address ?

update. second problem solved by using char* instead of IPAddress :slight_smile:

10

The sprintf is a matter of efficiency. I put it in one array, and send it as one packet. Every call to client.print() and client.println() creates another packet. Each packet has 48 bytes of overhead.

This sends the GET request in 7 packets.

client.print("GET ");
client.print(page);
client.println(" HTTP/1.1");
client.print("Host: ");
client.println(serverName);
client.println("Connection: close");
client.println();

My way sends it in 3 packets.

    sprintf(outBuf,"GET %s HTTP/1.1",page);
    client.println(outBuf);
    sprintf(outBuf,"Host: %s",serverName);
    client.println(outBuf);
    client.println(F("Connection: close\r\n"));

edit: And the most efficient way is in one packet. Insure outBuf is large enough to hold all this.

    sprintf(outBuf,"GET %s HTTP/1.1\r\nHost: %s\r\nConnection: close\r\n",page,serverName);
    client.println(outBuf);
11

And it's sill unstable.
I do afraid now the matter could be in the faultly shield or some restrictions on provider side.
Could anybody test my virtual server and if success post brief of working code.

server is at " www.promprojects.com"
it accepts with GET methods some data, you can try simly "www.promprojects.com\getdata.php".
you can also test to send some data like "http://www.promprojects.com/getdata.php?n=1&s=2&y=3&d=4&p=5"
on your browser you should get a message "DATA OK" returned by my php script there on.
But i failed to get any stable reply by sending requests from my Arduino...
Could anybody try ?

12

How often do you want the Arduino to send requests? I'm on pass 9 at 30 seconds between updates. How long does your code go before becoming unstable? A few minutes or a few days?

edit: Your server stalled at update 28. My code continued to run. Most sketches would lock up (freeze) there. Your server is also running again. It just missed the one response so far.

13

i need it to work stable for days being updated at least once in each two minutes.
Could you post your code?

14

Here is the code I am using. It is my client GET sketch from the playground modified for your requirements. I use dhcp to get an ip and dns to get your server ip.
http://playground.arduino.cc/Code/WebClient

/*
   Web client sketch for IDE v1.0.1 and w5100/w5200
   Uses GET method.
   Posted October 2012 by SurferTim
   Last modified June 17, 2013
*/

#include <SPI.h>
#include <Ethernet.h>

// this must be unique
byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };

// change to your network settings
IPAddress ip(192,168,2,2);
IPAddress gateway(192, 168, 2, 1);
IPAddress subnet(255, 255, 255, 0);

// change to your server
// IPAddress server(74,125,227,16); // Google

//Change to your domain name for virtual servers
char serverName[] = "www.promprojects.com";
// If no domain name, use the ip address above
// char serverName[] = "74.125.227.16";

// change to your server's port
int serverPort = 80;

EthernetClient client;
int totalCount = 0;
int loopCount = 0;
char pageAdd[32];

void setup() {
  Serial.begin(115200);

  // disable SD SPI
  pinMode(4,OUTPUT);
  digitalWrite(4,HIGH);

  // Start ethernet
  Serial.print(F("Starting ethernet..."));
//  Ethernet.begin(mac, ip, gateway, gateway, subnet);

  // If using dhcp, comment out the line above 
  // and uncomment the next 2 lines

  if(!Ethernet.begin(mac)) Serial.println(F("failed"));
  else Serial.println(F("ok"));
  digitalWrite(10,HIGH);

  Serial.println(Ethernet.localIP());

  delay(2000);
  Serial.println(F("Ready"));
}

void loop()
{
  if(loopCount < 30)
  {
    // if loopCount is less than 30, just delay a second
    delay(1000);
  }
  else
  {
    // every thirty seconds this runs
    loopCount = 0;

    // Modify next line to load different page
    // or pass values to server
    sprintf(pageAdd,"/getdata.php?n=%u",totalCount);

    // sprintf(pageAdd,"/arduino.php?test=%u",totalCount);

    if(!getPage(serverName,serverPort,pageAdd)) Serial.print(F("Fail "));
    else Serial.print(F("Pass "));
    totalCount++;
    Serial.println(totalCount,DEC);
  }    

  loopCount++;
}

byte getPage(char* ipBuf,int thisPort, char *page)
{
  int inChar;
  char outBuf[128];

  Serial.print(F("connecting..."));

  if(client.connect(ipBuf,thisPort))
  {
    Serial.println(F("connected"));

    sprintf(outBuf,"GET %s HTTP/1.1",page);
    client.println(outBuf);
    sprintf(outBuf,"Host: %s",serverName);
    client.println(outBuf);
    client.println(F("Connection: close\r\n"));
  } 
  else
  {
    Serial.println(F("failed"));
    return 0;
  }

  // connectLoop controls the hardware fail timeout
  int connectLoop = 0;

  while(client.connected())
  {
    while(client.available())
    {
      inChar = client.read();
      Serial.write(inChar);
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
    }

    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)
    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }

  Serial.println();

  Serial.println(F("disconnecting."));
  // close client end
  client.stop();

  return 1;
}

edit: Insure you change your serial monitor baud rate to 115200 for this code.

If you want to know what a server stall looks like on the serial monitor...
With no timeout, this will be the last entry on your serial monitor

connecting...connected

With my timeout code, you get this and it continues to run.

connecting...connected
Timeout
disconnecting

15

Seems the problem half-solved.
As already stated before - in my case there's a difference in connecting w5100 via router and via PC shared connection.
I've got for the time being really reliable connection, but only while connecting w5100 -> local ethernet card on my PC -> my PC's wi-fi -> router -> internet. This way everything works just perfect.
So, the only explanation is the problem is in connection itself, generally it could be two issues - my router can not handle w5100 in right manner (dns, gateway, own ip, router's ip on w5100 stated correctly). And the other possible issue is - some timings or buffers which are ok for local data transmission (from w5100 to local PC's ehternet card), but not enough for connection via router. At least i've obtained a way it works fine to remote server. Will try later with another router etc...

16

Are you using dhcp to get an ip and network settings on both the router and the PC? Or is one a static setup?

What is the make and model of router?

17

I use only static addressing which i'm sure is correct.
it's a D-Link DIR-300 hw rev A, flashed with fresh DD-WRT.