Using an Arduino to open a web page every minute

Hi to all, I've got a PHP script on a server and I need to open a specific page on that server every minute. The PHP script simply downloads images from some webcams and save them on the server (I simply want to build some stop motion videos of that webcams). Although there are a lot of web services (like mywebcron.com) specific for this purpose, I prefer using Arduino.

My program simply connects to the server, requests the page and wait some seconds before the next request. Problem: Arduino is not able to make repeated connections to a server. http://goo.gl/tOSnZ I tried to follow the instructions, but the last version of Arduino seems already to solve the problem (I'm using the OSX version), but it doesn't!! =(

So I decided to implement it in the "bad way", rebooting the Arduino after every request, using a command like this - http://goo.gl/LPsjc - :

void(* resetFunc) (void) = 0;

and calling resetFunc();

Well, it seems working, but every connection after the first one takes a lot of seconds, even some minutes, and after five or six connections, it doesn't work anymore...

May anyome help me to solve this problem?

Thanks :)

That earlier report sounds suspicious - a web server should not care what the incoming port number is, I suspect the original problem is a failure to closedown the socket cleanly. You open the connection to the webserver, you write a request, you flush your buffers and then read the reply fully to the end. Only when you have detected the server closing the connection should you close the socket at your end (in normal operation).

If specifying "HTTP 1.1" then you MUST use the header "Connection: close". If using "HTTP 1.0" then this header isn't needed or correct.

If you explicitly abort (close the client-side before the server side is ready) then you must drop the connection properly. If the ethernet library isn't sending the relevant FIN packets (and any required ACKs) this could be the issue here?

I think the Arduino based solution is not a sensible one for this problem.

All you need is a PC running a batch file that downloads the specified URL at the interval you want. You can either define the interval in your batch file, or have the batch file be called repeated by the task scheduler and save one frame each time it is called.

By the way, there are plenty of perfectly good and free software applications that you could run on the PC where the web cam is hosted to do all this for you without bothering with web pages and all that stuff. Is all this complexity really necessary? If feels as if you're creating layer upon layer of complexity for no apparent reason.

I don't see any reason to have an Arduino in this application.

However, can't really help unless you post your code.

Here is the code:

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xEF };
byte ip[] = { 192, 168, 10, 177 };
IPAddress server(..., ..., ..., ...); // website server IP address
String host = "www.myhost.com";
String page = "/mydir/index.php?src=arduino";
boolean waitingResponse = false;

int count=0;
int maxcount=1;

EthernetClient client;

void setup() {
  Serial.begin(9600);
  Ethernet.begin(mac, ip);
  Serial.println("Starting...");
}

void (*restart)(void) = 0;

void loop() {
  if(!waitingResponse) {
    if(count==maxcount) { // after one connection
      Serial.println("Sleeping for 50 seconds");
      delay(50000);
      Serial.println("Restarting...");
      restart();
      return;
    }
    connectToServer();
    count++;
  }
  else {
    processResponse();
  }
}

void connectToServer() {
  if(waitingResponse) return;
  
  Serial.println("Connecting...");
  
  int tries = 0;
  while (!client.connect(server, 80)) {
    delay(200);
    tries++;
    if (tries > 100) { // ok we have problem
      Ethernet.begin(mac, ip); // solve it by brute force
      tries=0;
    }
  }
  client.print("GET ");
  client.print(page);
  client.println(" HTTP/1.0"); // here I can adjust as suggested, using HTTP/1.1 and force a connection close
  client.print("Host: "); // needed in my case (virtualserver)
  client.println(host);
  client.println();
  waitingResponse = true;
}

boolean processResponse() {
  if(!waitingResponse) return true;
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }
  
  if (!client.connected()) {
    Serial.println();
    Serial.println("Disconnecting.");
    client.stop();
    
    waitingResponse = false;
    return false;
  }
  
  return true;
}

I know that it’s possible to use a script to load the page every minutes, with curl or wget, and I did.
But I prefer using Arduino simply because it doesn’t take me to keep a computer always on (only for this purpose) and it’s cheaper in terms of power supply.

I still haven’t tried the HTTP/1.1 solution, but I will tonight or maybe tomorrow. :grin:
Thanks to everyone! :slight_smile:

I've just tried the HTTP/1.1 version with Connection:close and it doesn't work =(

auino: But I prefer using Arduino simply because it doesn't take me to keep a computer always on (only for this purpose) and it's cheaper in terms of power supply.

Well the server with the webcam you are using is already running, isn't it? So why not just setup a cron job on that system to run the script or take the picture or whatever? That's even less power wasted than using an arduino.

No, the server is located on another city and I have only some web space there, plus services like FTP access or email.

auino: No, the server is located on another city and I have only some web space there, plus services like FTP access or email.

I'd get a better web host in that case.

I don't understand why you split your "connect" code and "response" code. I would have just taken the simple ethernet example and created a function to open the connection, send the query, and close the connection.

Maybe you evolved from this already?

byte mac[] = { 
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 
  10, 0, 0, 177 };
byte server[] = { 
  64, 233, 187, 99 }; // Google

EthernetClient client;

void setup() {
  Ethernet.begin(mac, ip);
  Serial.begin(9600);
}

void loop() {
  connectToServer();
  delay(50000);
}

void connectToServer() {
  // this was setup...
  delay(1000);
  Serial.println("connecting...");

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /search?q=arduino HTTP/1.0");
    client.println();
  }  else {
    Serial.println("connection failed");
    return;
  }

  // this was loop...
  while (!client.available()) {
    char c = client.read();
    Serial.print(c);
  }
  // delay(1000); // maybe add a delay to allow the connection to close.
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
  }  
  return;
}

Thanks, but your solution doesn't work, at least for me... Indeed, after the first connection (or some connections), the Arduino can't establish the connection anymore... =(

auino: No, the server is located on another city and I have only some web space there, plus services like FTP access or email.

You've got a [u]hosted[/u] webcam???

Is the web page you want to hit also hosted in the same place?

How does the web page save your snapshot, and where?

Is the web page under your control?

Oh, and how frequently do you expect to access it?

You've got a hosted webcam??? No, the webcams are some public webcams on the web.

Is the web page you want to hit also hosted in the same place? No, the PHP script is on a different server.

How does the web page save your snapshot, and where? The PHP script downloads the webcam images and stores them into a specific directory on the server. Then I can download all the images via FTP.

Is the web page under your control? Yes, it's a PHP script and I can change it.

I'd have thought that capturing the webcam images on a PC would be the simplest and best solution. You can always sleep the PC when it's inactive.

Below is some client test code that seems to be fairly stable. You could try including some type of timing code in it to have it make a request to your server at the desired interval.

//zoomkat 12-08-11
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address
byte ip[] = { 192, 168, 1, 102 }; // ip in lan assigned to arduino
//byte gateway[] = { 192, 168, 1, 1 }; // internet access via router
//byte subnet[] = { 255, 255, 255, 0 }; //subnet mask
byte myserver[] = { 208, 104, 2, 86 }; // zoomkat web page server IP address
EthernetClient client;
//////////////////////

void setup(){

  Ethernet.begin(mac, ip);
  //Ethernet.begin(mac, ip, subnet, gateway);
  Serial.begin(9600); 
  Serial.println("Better client test 12/01/11"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(myserver, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0"); //download text
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}

@zoomcat, thanks but your code it's pretty similar to mine... so it shouldn't work =(

This is an Arduino problem, I think... it simply can't manage this... The PC solution is not a solution. I'm in an Arduino forum and I'm trying to solve this problem with my Arduino!!

Your read loops are different, give zoomkats code a proper look.

You try and read while the client is unavailable.

  // this was loop...
  while (!client.available()) {
    char c = client.read();
    Serial.print(c);
  }

this waits until...

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available

If you want to use a hammer to demolish a building, don't blame the tools. :)