Web Client not working consistently

I’m still an Arduino newbie (and new to programming in general), so please bear with me.

I’m having a recurring problem of not getting updates of sensor readings posted to a web server. It will work as expected the first time, sometimes a few times, then it stops working right. The Arduino suggests that it thinks everything is getting updated successfully, but the web server isn’t actually getting everything. It posts two variables to the web server by hitting a php page, which sticks them into a database and sends me an email so I can easily see the update happened as expected.

The overall sketch is much bigger than what I’ve posted below, but this is the part that actually handles the web update. The rest involves the sensor, the LCD screen, and getting (and updating) the time from the web.

My big question is: why does it update the first time, or first few times, but then stop working right?

Is there something additional I need to do? What is client.flush(), and would it do anything for me? Should I add that to the sketch? Is there something else I’m missing? I’ve searched through a lot of threads for similar projects and can’t see anything different about what others are doing (there are a lot of temperature sensor projects, and a few greenhouse temp/humidity projects).

long int deviceID = 100002;      // set the unique device ID used to update the remote database


void updateDB()  {
    
  // if there's incoming data from the net connection.
  // send it to the LCD.  This is for debugging
  // purposes only:
  if (client.available()) {
    char clientRead = client.read();
  }

 if (currentTime - lastConnectionTime > 120000)  {
    dbUpdated = false;
  }
  
  
  if (hour(locNow) == 9 && minute(locNow) == 0)  {                            // update the remote DB daily at 9:00 Local time.
      dbClientConnect();
  }
  
  if (previousUpdate == 0 && currentTime > 120000)  {                            // update the remote DB daily at 16:30 Local time.
    dbClientConnect();
  }
  
  if (successfulUpdates < 28 && hour(locNow) % 2 == 0 && minute(locNow) == 0)  {
      dbClientConnect();
  }
}  


void dbClientConnect()  {
 if (!dbUpdated)  { 
  if(!client.connected())  {
    if (client.connect(server, 80)) {
      client << "GET /submit.php?deviceID=" << deviceID << "&sensor_reading=" << sensorReading << " HTTP/1.1" << endl;
      client << "Host: myserver.com" << endl;
      client << endl;
      
      dbUpdated = true;
      updateStatus = 1;
      dbAttempts = 0;
      previousUpdate = locNow;
      lastConnectionTime = millis();
      successfulUpdates++;
      client.stop();
    }
    else {
      updateStatus = 2;
      dbUpdated = false;
      if (dbAttempts > 0) client.stop();
      failedAttempts++;
      failedAttemptsTotal++;
    }
  }
  // lastConnected = client.connected();
 }
  
}

You are probably running out of sockets. You are probably leaving characters in the rx buffer. If you do, the socket will not close correctly. The w5100 has only 4 sockets.

Here is my code. It reads and displays the server response on the serial monitor and has a timeout feature that will prevent lockups.
http://playground.arduino.cc/Code/WebClient

SurferTim: You are probably running out of sockets. You are probably leaving characters in the rx buffer. If you do, the socket will not close correctly. The w5100 has only 4 sockets.

Here is my code. It reads and displays the server response on the serial monitor and has a timeout feature that will prevent lockups. http://playground.arduino.cc/Code/WebClient

Is it as easy as adding client.flush(); into the mix between updates to the web server? If not, what is it in your code that clears out the buffer and closes the socket?

What frustrates me is that I was having trouble with this before, and then added the client.stop(); and it seemed to fix things. As a trial, I set it up to update the server every 10 minutes. It ran for half a day quite well, loading up my database with far more updates than I needed. I changed it to daily and then it only ran for a week (7 daily updates) before it stopped working as expected again. Now I can't get more than the initial update, or maybe 2 or 3 more after that, less than the 7 I got before.

Is it as easy as adding client.flush(); into the mix between updates to the web server?

No. That will flush only one packet. What if the response is several packets long? The server will close the connection on its end when finished. Then you close your end. If the connection breaks (hardware fail), the 10 second timeout will abort the loop.

  int connectLoop = 0;

  // stay in this next loop until the server closes the connection
  while(client.connected())
  {
    // stay in this next loop until a received packet is read
    // and removed from the rx buffer
    while(client.available())
    {
      inChar = client.read();
      Serial.write(inChar);
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
    }

    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)
    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }

  Serial.println();

  Serial.println(F("disconnecting."));
  // close client end
  client.stop();

Something like this?

long int deviceID = 100002;      // set the unique device ID used to update the remote database


void updateDB()  {
    
  // if there's incoming data from the net connection.
  // send it out the serial port.  This is for debugging
  // purposes only:
  if (client.available()) {
    char clientRead = client.read();
  }
  
  if (currentTime - lastConnectionTime > 120000)  {
    dbUpdated = false;
  }
  
  
  if (hour(locNow) == 9 && minute(locNow) == 0)  {                            // update the remote DB daily at 9:00 Local time.
      dbClientConnect();
  }
  
  if (previousUpdate == 0 && currentTime > 120000)  {                            // update the remote DB 2 minutes after the sketch starts.
    dbClientConnect();
  }
  
  if (successfulUpdates < 28 && hour(locNow) % 2 == 0 && minute(locNow) == 0)  {      // For the first 28 updates, update the remote DB every 2 hours.
      dbClientConnect();
  }
}  


void dbClientConnect()  {
  if (!client.connected() && lastConnected) {
    client.stop();
  }
  
  if (!dbUpdated)  { 
    if(!client.connected())  {
      if (client.connect(server, 80)) {
        client << "GET /submit.php?deviceID=" << deviceID << "&sensor_reading=" << sensorReading << " HTTP/1.1" << endl;
        client << "Host: myserver.com" << endl;
        client << endl;
        
        ///////////////////////////************Begin SurferTim's Code*****************************
        
        int connectLoop = 0;

        // stay in this next loop until the server closes the connection
        while(client.connected())  {
          // stay in this next loop until a received packet is read
          // and removed from the rx buffer
          while(client.available())  {
            int inChar = client.read();
            Serial.write(inChar);
            // set connectLoop to zero if a packet arrives
            connectLoop = 0;
          }

          connectLoop++;

          // if more than 10000 milliseconds since the last packet
          if(connectLoop > 10000)  {
            // then close the connection from this end.
            Serial.println();
            Serial.println(F("Timeout"));
            client.stop();
          }
          // this is a delay for the connectLoop timing
          delay(1);
        }

        Serial.println();
        Serial.println(F("disconnecting."));
        // close client end
        client.stop();
        
        
        ///////////////////////////************End SurferTim's Code*****************************
        
        dbUpdated = true;
        updateStatus = 1;
        dbAttempts = 0;
        previousUpdate = locNow;
        lastConnectionTime = millis();
        successfulUpdates++;
      }
      else {
        updateStatus = 2;
        dbUpdated = false;
        // if (dbAttempts > 0) client.stop();
        client.stop();
        dbAttempts++;
        dbAttemptsTotal++;
      }
    }
  }
  
  lastConnected = client.connected();
}

I would get rid of this. Until you connect to the server, client is not valid.

  // if there's incoming data from the net connection.
  // send it out the serial port.  This is for debugging
  // purposes only:
  if (client.available()) {
    char clientRead = client.read();
  }

The rest looks ok. How does it run?

SurferTim:
The rest looks ok. How does it run?

I’ll let you know after letting it run for a while. Just to get a better idea quicker, I changed this:

if (successfulUpdates < 24 && hour(locNow) % 2 == 0 && minute(locNow) %10 == 0)  {
      dbClientConnect();
  }

to this:

if (successfulUpdates < 25 && minute(locNow) %10 == 0)  {
      dbClientConnect();
  }
  
  if (24 < successfulUpdates < 38 && hour(locNow) % 4 == 0 && minute(locNow) == 0)  {
      dbClientConnect();
  }

Thanks for the help, I really appreciate it. I’ll let you know how it works for me.

No problem! Glad I could help. I recommend watching the serial monitor for a while. If you ever see "Timeout" during a connection, the Arduino would have locked up at that point.

  if (24 < successfulUpdates < 38

Nope.

SurferTim:
I recommend watching the serial monitor for a while. If you ever see “Timeout” during a connection, the Arduino would have locked up at that point.

I’ll have to move it to do that. The sensor is hard wired into a tank to watch the material level. It’s nowhere near the computer. I use an LCD display instead.

Arrch:

  if (24 < successfulUpdates < 38

Nope.

Yep. The statement above that one gets it up to 25, this one takes it from 25 to 37. Don’t let the misaligned tab throw you off, they are separate statements, not nested ones.

The purpose is this: for the first 25 updates, update every 10 minutes. From update 25 to 37, update every 4 hours. After that, update daily at 9:00AM local time. Once this thing is solid as a rock, I’ll remove those two statements and it will only update daily, which is often enough for the material I’m monitoring.

SurferTim: I recommend watching the serial monitor for a while.

I need one of these.

I do stuff like that before deploying a system, but if it is already operational when you find the bug, use the LCD to display the number of timeouts if you think it may be causing a delay in your data transfers. But if it is transferring data and doesn't lock up, which it shouldn't, you are good to go.

SurferTim:
The rest looks ok. How does it run?

It looks like I had some problems overnight. It ran every 10 minutes until 9:40PM as expected, the 25 at 10 minute intervals I programmed in. Then it didn’t go again until 4:00, so it missed the 12:00 update (if I understand the % function, hour 0 % 4 == 0 should be true). Even if I’m wrong that 0/4 should leave a 0 remainder, it just missed the 8:00 update. The successfulUpdates reads 28 on the display, so it’s definitely within the 24 < sU <38 range. I have about 40 minutes before I expect the daily 9:00AM update to trigger, but am not very confident at this point.

Despite nothing in the database for the missing times, the LCD says it’s going through the code to update. Right now: last update 8:00AM

I did not evaluate your logic that controls the time the sketch tries the connection. You may need to add more LCD code now to show the reason for the fails.

GOOD NEWS!! It did not lock up due to the ethernet code. :)

I don't think I've ever seen it lock up totally, it just doesn't get the query out to the website.

What really confuses me is that it always seems to work if the time between updates is measured in minutes. Once it gets to hours (or days), I start having problems.

Now you must figure out if it is

  1. the timing code that is not running the ethernet client code,
  2. the w5100 is not making a connection to the server,
    or
  3. the connection is timing out after establishing the connection.

SurferTim: Now you must figure out if it is 1) the timing code that is not running the ethernet client code, 2) the w5100 is not making a connection to the server, or 3) the connection is timing out after establishing the connection.

I'm fairly certain it isn't 1), and am not sure exactly how to go about determining if it's 2) or 3). The LCD will tell me the last time it ran the ethernet client code, and it always displays the last time I expected it to.

I'm going to try setting it up with a second sensor near a computer so I can run the serial monitor. The LCD can help, but it doesn't update fast enough at times, and at other times it does update, clearing out something I may have wanted to see to display something else.

Maybe you should display a count of the number of missed and timed out connections. Here is how to total them up. You work out the way you want to display failCount and timeoutCount on the LCD.

// in global declarations
int timeoutCount = 0;
int failCount = 0;

// then later in the connection attempt
     if (client.connect(server, 80)) {
       // your connection success stuff
     } 
     else {
        Serial.println(F("Connection failed"));
        failCount++;
     }

// and in the timeout code     
          if(connectLoop > 10000)  {
            // then close the connection from this end.
            Serial.println();
            Serial.println(F("Timeout"));
            timeoutCount++;
            client.stop();
          }

Canyonero: I have about 40 minutes before I expect the daily 9:00AM update to trigger, but am not very confident at this point.

The 9:00AM worked, but just for peace of mind, I'm still going to move it over to a computer and use a second sensor so I can use the Serial monitor.

That is a good sign for the ethernet code. Now to find what is failing. The connection attempt or the timeout.