Ethernet client cannot connect following connection errors

system · January 5, 2012, 10:33am

Hello All,

I am using an Arduino Uno with Ethernet Shield on ARDUINO1.0. I am writing some logging software, which continuously sends HTTP requests, but after 4 connections errors (Errors appear to occur randomly; I believe these to be be network problems), the Arduino can no longer connect to the server at all (Infinite loop of connection errors).

I recall somewhere mentioning maximum open connections being 4, but I believe that I correctly close the connections each time. Please do correct me if I am wrong.

#include <SPI.h>
#include <Ethernet.h>

EthernetClient client;

char* testMessage = "{\"test\":\"test\"}";

byte mac[] = {0x00,0x23,0xDF,0x82,0xD4,0x01};
IPAddress server({redacted});
byte localIP[] = {155,198,114,4};
int port = {redacted};

void setup(){
  Serial.begin(9600);  
  Ethernet.begin(mac, localIP);
  
  delay(1000);
  
  Serial.println("Arduino Started");
  
}

void loop(){
  
  httpPut(testMessage, "/topic/:test/put?sessionId=SESpbn4u1g5k&expiry=30s", "201");
  
}

/* Does a HTTP PUT request to the server */
bool httpPut(const char* data, const char* url, char* expectedStatusCode){
  if (client.connect(server, port)){
    client.print("PUT");
    client.print(" ");
    client.print(url);
    client.print(" ");
    client.println("HTTP/1.0");
    
    client.print("Content-length: ");
    client.println(strlen(data));
    
    client.println();
    client.println(data);
    client.println();
    client.println();
    
    while (!client.available()){
      delay(1); 
    }
    
    if (checkStatusCode(expectedStatusCode)){
      while(client.available()){
        char c = client.read();
      }
    }else{
      Serial.println(F("Error in httpPut(...), returned unexpected error code"));
      client.stop();
      client.flush();
      if (client.connected()){
        Serial.println(F("Error: Client still connected after calling stop()"));
      }
      return false;
    }
    
  }else{
    Serial.println(F("Error in httpPut(...), could not connect"));
    client.stop();
    client.flush();
    if (client.connected()){
      Serial.println(F("Error: Client still connected after calling styop()"));
    }
    return false;
  }
  client.stop();
  client.flush();
  if (client.connected()){
    Serial.println(F("Error: Client still connected after calling styop()"));
  }
  return true;
}

/* Check response status code from server. Used by internal HTTP request functions. */
bool checkStatusCode(char* expected){
  
  bool status = true;
  
  // Read the HTTP version
  for (int i=0;i<9;i++){
    char c = client.read();
  }
  
  for (int i = 0;i<3;i++){
    char c = client.read();
    
    if (c != expected[i]){
      return false;
    }
  }
  
  return true;
  
}

The Serial output:

Arduino Started
Error in httpPut(...), could not connect
Error in httpPut(...), could not connect
Error in httpPut(...), could not connect
Error in httpPut(...), could not connect

← After this point, connection error loops infinity.

Error in httpPut(...), could not connect
Error in httpPut(...), could not connect
Error in httpPut(...), could not connect

Hopefully this is a trivial problem, but any help would be appreciated.

system · January 5, 2012, 12:34pm

    if (checkStatusCode(expectedStatusCode)){
      while(client.available()){
        char c = client.read();
      }
    }

You made a request, and the server sent back a response. If the status code was not 201, you shuffle the response off to the bit bucket. Perhaps there was a clue to the problem in that response.

If it was 201, you call:

  client.stop();
  client.flush();
  if (client.connected()){
    Serial.println(F("Error: Client still connected after calling styop()"));
  }

It's really time that you looked at the source code for the EthernetClient class. Once stop() is done, there is nothing to flush, and the client can not possibly still be connected.

You should be, I think, reading the server response, and then stopping the client. If you REALLY don't want to read the server response, call flush() first, then stop().

system · January 6, 2012, 1:10pm

Thanks PaulS for your reply,

I agree with you regarding keeping the status code, and this is something which I will make sure is implemented later.

Correct me if I’m wrong, but since all the errors are of the form Error in httpPut(...), could not connect, these are not errors due to unexpected status codes, but due to being unable to connect?

I had a look through the Ethernet code as suggested, and have commented and put some Serial.println(...)s in the connect(...) method so at runtime I can see exactly why it was unable to connect. Please tell me if any of the comments in the following code are wrong.

int EthernetClient::connect(IPAddress ip, uint16_t port) {
  //Check to see if there is an open socket on this instance of EthernetClient
  if (_sock != MAX_SOCK_NUM){
    Serial.println("Err 1 - Socket already open on this EthernetClient instance");
    return 0;
  }

  //Tries to find an available socket
  for (int i = 0; i < MAX_SOCK_NUM; i++) {
    uint8_t s = W5100.readSnSR(i);
    if (s == SnSR::CLOSED || s == SnSR::FIN_WAIT) {
      _sock = i;
      break;
    }
  }

  //_sock will eq MAX_SOCK_NUM if no sockets were available
  if (_sock == MAX_SOCK_NUM){
    Serial.println("Err 2 - All sockets already in use by Ethernet card");
    return 0;
  }

  //Open TCP Socket
  _srcport++;
  if (_srcport == 0) _srcport = 1024;
  socket(_sock, SnMR::TCP, _srcport, 0);

  // Establish (Active) TCP connection
  if (!::connect(_sock, rawIPAddress(ip), port)) {
    _sock = MAX_SOCK_NUM;
	Serial.println("Err 3 - Cannot cestablish active connection");
    return 0;
  }

  //Checks for the connection to be established
  while (status() != SnSR::ESTABLISHED) {
    delay(1);
    if (status() == SnSR::CLOSED) {
      _sock = MAX_SOCK_NUM;
	  Serial.println("Err 4 - Connection closed by server");
      return 0;
    }
  }

  return 1;
}

I also did some minor changes to the code (mostly in relation to PaulS’s comment regarding flushing after stopping)

#include <SPI.h>
#include <Ethernet.h>

EthernetClient client;

char* testMessage = "{\"test\":\"test\"}";

byte mac[] = {0x00,0x23,0xDF,0x82,0xD4,0x01};
IPAddress server({redacted});
byte localIP[] = {155,198,114,4};
int port = {redacted};

void setup(){
  Serial.begin(9600);  
  Ethernet.begin(mac, localIP);
  
  delay(1000);
  
  Serial.println("Arduino Started");
  
}

void loop(){
  
  httpPut(testMessage, "/topic/:test/put?sessionId=SESg86tvt9pd&expiry=30s", "201");
  
}

/* Does a HTTP PUT request to the server */
bool httpPut(const char* data, const char* url, char* expectedStatusCode){
  if (client.connect(server, port)){
    client.print("PUT");
    client.print(" ");
    client.print(url);
    client.print(" ");
    client.println("HTTP/1.0");
    
    client.print("Content-length: ");
    client.println(strlen(data));
    
    client.println();
    client.println(data);
    client.println();
    client.println();
    
    while (!client.available()){
      delay(1); 
    }
    
    if (checkStatusCode(expectedStatusCode)){
      while(client.available()){
        char c = client.read();
      }
    }else{
      Serial.println(F("Error in httpPut(...), returned unexpected error code"));
      client.flush();
      client.stop();
      return false;
    }
    
  }else{
    Serial.println(F("Error in httpPut(...), could not connect"));
    client.stop();
    return false;
  }
  
  client.stop();
  return true;
}

/* Check response status code from server. Used by internal HTTP request functions. */
bool checkStatusCode(char* expected){
  
  bool status = true;
  
  // Read the HTTP version
  for (int i=0;i<9;i++){
    char c = client.read();
  }
  
  for (int i = 0;i<3;i++){
    char c = client.read();
    
    if (c != expected[i]){
      return false;
    }
  }
  
  return true;
  
}

Now having run this, the output I get is as follows:

Arduino Started
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Err 2 - All sockets already in use by Ethernet card
Error in httpPut(...), could not connect
Err 2 - All sockets already in use by Ethernet card
Error in httpPut(...), could not connect
Err 2 - All sockets already in use by Ethernet card
Error in httpPut(...), could not connect

We can see that the first 4 errors are due to the connection being closed by the server (these occur at random intervals, assumed to be random network errors). After the 4th error, the Arduino loops infinity, being unable to connect as Err 2 - All sockets already in use by Ethernet card. This suggests to me that something is not closing the ports correctly?

Again, please tell me if any of my assumptions are wrong.

Hopefully this will turn out to be a trivial error.
Thanks again for your time, it’s really appreciated!

system · January 6, 2012, 3:16pm

Not sure if this helps, but i’ve added a few more Serial prints to the Ethernet code segment below to try and see what is happening internally:

  //Tries to find an available socket
  for (int i = 0; i < MAX_SOCK_NUM; i++) {
    uint8_t s = W5100.readSnSR(i);
	Serial.print("Port ");
	Serial.print(i);
	Serial.print(" = ");
	Serial.println(s, HEX);
    if (s == SnSR::CLOSED || s == SnSR::FIN_WAIT) {
      _sock = i;
      break;
    }
  }

Which gives the following output

Arduino Started
Port 0 = 0
Port 0 = 0
Port 0 = 0
...
Port 0 = 0
Port 0 = 0
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Port 0 = 17
Port 1 = 0
Port 0 = 17
Port 1 = 0
Port 0 = 17
Port 1 = 0
...
Port 0 = 17
Port 1 = 0
Port 0 = 17
Port 1 = 0
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Port 0 = 17
Port 1 = 17
Port 2 = 0
Port 0 = 17
Port 1 = 17
Port 2 = 0
...
Port 0 = 17
Port 1 = 17
Port 2 = 0
Port 0 = 17
Port 1 = 17
Port 2 = 0
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 0
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 0
...
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 0
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 0
Err 4 - Connection closed by server
Error in httpPut(...), could not connect
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 17
Err 2 - All sockets already in use by Ethernet card
Error in httpPut(...), could not connect
Port 0 = 17
Port 1 = 17
Port 2 = 17
Port 3 = 17
Err 2 - All sockets already in use by Ethernet card
Error in httpPut(...), could not connect
...

Later on they one by one go to 1C.

From the SnSR class, we can see:

static const uint8_t ESTABLISHED = 0x17;
static const uint8_t CLOSE_WAIT  = 0x1C;

So we can see that the ports are remaining established after a failed connect, even though stop( ) is called. Any suggestions?

zoomkat · January 6, 2012, 3:34pm

The below client test code has some code suggested by forum members to enhanse the connection with a server.

//zoomkat 12-08-11
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////

void setup(){

  Ethernet.begin(mac, ip);
  //Ethernet.begin(mac, ip, subnet, gateway);
  Serial.begin(9600); 
  Serial.println("Better client test 12/01/11"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(myserver, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

system · January 6, 2012, 3:57pm

I can do simple HTTP requests (GET, PUT) fine; the HTTP errors only appear after a very large number of requests (as indicated above, after a client.connect(...) error, that port stays Established for some reason).

The code you suggest does not appear to do any further tests or "enhanse the connection with a server", and your sendGET(...) method does essentially the same as my httpPut(...) (but without checking the HTTP status code). Am I missing something?

SurferTim · January 6, 2012, 4:07pm

I think zoomkat means that the server closes the connection (with HTTP/1.0).

while(client.connected() && !client.available()) delay(1); //waits for data while (client.connected() || client.available()) { //connected or data available char c = client.read(); //gets byte from ethernet buffer Serial.print(c); //prints byte to serial monitor }

I think your error checking is better tho! :)

zoomkat · January 6, 2012, 4:16pm

Am I missing something?

You are aware that your code eventually crashes, so you are on top of things. What is the "very large number of requests (as indicated above," of which you speak? tens, hundreds, thousands?

system · January 6, 2012, 4:28pm

Surely the bolded bit is wrong?

while(client.connected() && !client.available()) delay(1); //waits for data while (client.connected() || client.available()) { //connected or data available char c = client.read(); //gets byte from ethernet buffer Serial.print(c); //prints byte to serial monitor }

Why do a client.read() if client.connected() is true but client.available() is false? You're reading data which is not available?

zoomkat: What is the "very large number of requests (as indicated above," of which you speak? tens, hundreds, thousands?

Thousands

SurferTim · January 6, 2012, 4:36pm

I do not use that exact code. Mine is more like this:

while(client.connected() && !client.available()) delay(1); //waits for data

while (client.connected()
{
   // stay in this loop until the server closes this connection
   if(client.available()) { 
      // If any characters are available, get those. The connection will not close until the rx buffer is empty
      char c = client.read(); //gets byte from ethernet buffer
      Serial.print(c); //prints byte to serial monitor
   }
}

The server may stall during the download if the webpage is dependent on a database (SQL/PHP). There is a possibility the server will send the page header, then access a database. You may empty the receive buffer before the server can access the database.

Just from experience: If there is anything in the rx buffer, the socket will not close.

system · January 6, 2012, 4:53pm

SurferTim: Just from experience: If there is anything in the rx buffer, the socket will not close.

Ahh, that makes sense. I've changed my code to the following

    while (client.connected() && !client.available()){
      delay(1); 
    }
    
    if (checkStatusCode(expectedStatusCode)){
      while (client.connected()){
        client.flush();
      }
      
      client.stop();
      return true;
    }else{
      while (client.connected()){
        client.flush();
      }
      
      client.stop();
      Serial.println(F("Error in httpPut(...), returned unexpected error code"));
      return false;
    }

That should ensure the rx buffer is empty. (Not the most elegant of code, but it can be cleaned later)

Unfortunately it still fails as ports are not being closed after a client.connect(...) failure. After the 4th connect() error, all ports are either ESTABLISHED or CLOSE_WAIT. Any other ideas?

Thanks again for your support guys, really appreciated.

SurferTim · January 6, 2012, 4:56pm

One more idea. Delete any mention of client.flush(). I don't use it, and don't recall ever needing it.

zoomkat · January 6, 2012, 5:39pm

Why do a client.read() if client.connected() is true but client.available() is false? You're reading data which is not available?

I think the read is made if the client is still connected or if there is still data in the input buffer after the connection may have been closed. It sees to have solved a problem with occasional missing data from the server.

system · January 6, 2012, 6:15pm

SurferTim:
One more idea. Delete any mention of client.flush(). I don’t use it, and don’t recall ever needing it.

I have simplified my code some more to try and work out where the error may lie

#include <SPI.h>
#include <Ethernet.h>

EthernetClient client;

char* testMessage = "{\"test\":\"test\"}";

byte mac[] = {0x00,0x23,0xDF,0x82,0xD4,0x01};
IPAddress server({redacted});
byte localIP[] = {155,198,114,4};
int port = {redacted};

int i = 0;

void setup(){
  Serial.begin(19200);  
  Ethernet.begin(mac, localIP);
  
  delay(1000);
  
  Serial.println("Arduino Started");
  
}

void loop(){
  
  httpPut(testMessage, "/topic/:test/put?sessionId=SESs9qg7g242&expiry=30s");
  
}

/* Does a HTTP PUT request to the server */
bool httpPut(const char* data, const char* url){
  if (client.connect(server, port)){
    client.print("PUT");
    client.print(" ");
    client.print(url);
    client.print(" ");
    client.println("HTTP/1.0");
    
    client.print("Content-length: ");
    client.println(strlen(data));
    
    client.println();
    client.println(data);
    client.println();
    client.println();
    
    while (client.connected() && !client.available()){
      delay(1); 
    }
    
    while(client.available() || client.connected()){
      char c = client.read();
      Serial.print(c);
    }
    
    client.stop();
    return true;
    
  }else{
    Serial.println(F("Error in httpPut(...), could not connect"));
    client.stop();
    return false;
  }
}

Unfortunately the same problem of ports being established exists
This is getting rather frustrating.

system · January 6, 2012, 6:35pm

This couldn't be caused by anything on the server? I'm right in saying that client.stop() (with its graceful and forceful closing of ports) would would deal with that?

zoomkat · January 6, 2012, 7:14pm

This couldn't be caused by anything on the server?

The client connection is closed before the server is finished for some reason, then the server might not establish another connection thinking one already exist. How fast are the repeated connections to the server being made?

SurferTim · January 7, 2012, 12:17pm

Sochaduino: This couldn't be caused by anything on the server? I'm right in saying that client.stop() (with its graceful and forceful closing of ports) would would deal with that?

What is your recent experience telling you? Since you refer to graceful and forceful, you have seen the ethernet library code, correct? Maybe a second is not long enough to send the FIN to the server. Have you tried increasing that wait period before forcing a close?

I have tried repeated client downloads from an Apache server with good results. I stopped the test just short of 5000 downloads, but I was waiting 20 seconds between requests.

By repeated connection attempts, it is possible the server may think you are attempting a DoS attack. My router can "tarpit" connection attempts like that. That is designed to do just what you are describing.

Just a thought...

system · January 16, 2012, 10:37am

Hello again (Sorry for taking so long to get back to this thread; its been a busy week!)

zoomkat: How fast are the repeated connections to the server being made?

The connections are as fast as the arduino can go. It look like ~25 connections per second on the main server I am testing with.

SurferTim: What is your recent experience telling you? Since you refer to graceful and forceful, you have seen the ethernet library code, correct? Maybe a second is not long enough to send the FIN to the server. Have you tried increasing that wait period before forcing a close?

I've done some more testing, and it appears that the Arduino never uses a forceful closing of the socket (it always returns after the graceful closing)

SurferTim: By repeated connection attempts, it is possible the server may think you are attempting a DoS attack. My router can "tarpit" connection attempts like that. That is designed to do just what you are describing.

I don't think this is the problem as we've done various stress tests before on the server, and a previous version of the sensor appears to be running fine. Something I should perhaps look into.

As a quick summary of everything so far:

Arduino network sensor posts a JSON string to the server ~25 time per second
HTTP status code returns 2XX
All data in rx is read/flushed
Occasionally there is a random network error which means that client cannot connect. An error is printed to Serial, and client.stop() is called
After each of these errors, that socket remains open (ESTABLISHED) despite having called client.stop(), resulting in another socket being used
All calls client.stop() close the socket gracefully
Once 4 errors have occurred, all sockets are in 'use' and the Arduino can no longer connect to the server

This is a list of things I think have been established so far (although not necessarily correct). I'm going to discuss with some people at my end if it could caused to to our internal network or servers. I'll post back with anything I find.

In the mean time, if there are any more ideas, these are very welcome. I feel like if i don't solve this, its going to come back and bite me later! Thanks again for all your help :-)

SurferTim · January 16, 2012, 11:01am

If I want to transfer stuff that often, I use a persistent connection. I leave the connection open. I use a custom server on the computer tho.

I use mine for joystick controlled servos. I haven't ran it for more than 8 hours or so at a time. Don't know how it does after that. It has no problem with temporary network fails. Disconnects and reconnects gracefully.

system · May 6, 2021, 10:36am