True two way communication between Arduino server and PHP server

tl;dr: I want to be able to push a button connected to my Arduino and turn on an output and at the same time be able to click a button on a self hosted website and turn off that same output. Sounds simple enough.

Long version:

I’m working on a home automation project (yes, another one) using the Arduino and an Ethernet shield (lots of Arduinos actually). This one, though, is much more advanced than all the examples I’ve been able to find and I need a bit of help getting around a certain limitation of the Arduino.
I’ve attached a diagram of my setup to give you an idea of what I’m working with.

Web Server

I have a PHP REST server set up that is connected to a database and also an HTML frontend. For those of you not familiar with REST, essentially in my case all I have to do to get data from the database is send a GET request to a certain URL with parameters and in order to write information to the database I just have to send a POST request to the same URL. My REST server differentiates between the various types of requests and performs the necessary database operations. The HTML interface updates via AJAX and is what will be sending information to the Arduino (using javascript/AJAX calls).

Arduino Server/Client

What I’m trying to do on the Arduino is have it so that the Arduino server can receive information from my server and at the same time be able to send data to my server. I have successfully implemented both of these functions separately but I am having a hard time putting them together.

The logical approach to this is to have two functions in the main loop repeating, one which initializes a client to send information and another that initializes a server to receive information. If there is nothing to send or receive then the loop continues on as usual. This would work except for the fact that initializing a server pauses the loop until it receives information EthernetClient client = server.available(); //This pauses the loop This makes the idea pointless because as soon as the server is initialized, I can’t send any information to the server when a button is pushed, etc because the program has stopped at the server. Essentially this becomes long polling but I can only have 1 connection at a time. (see code below)

void loop() {
  // Happy loop
  if (someone has pushed a button)  {
      sendPOST(); // Send the data to the server. This is the client
    }
  
EthernetClient client = server.available();  //This is what is hanging the loop. The server.

//I want to be able to do this. If there is no client connected to the server then continue on with
//the loop, but I never get to this point because the above code stops the loop
if (!client) { 
  Serial.println("not connected");
  client.stop();
  }  
//If there is a client
  if (client) {
 //receive the information
}

} //end loop.

So such is my dilemma. Is it possible to truly have a server and client rotating on the Arduino without the server stopping the loop until it gets something? And can someone offer me a better solution to this?

Solutions I’m considering:

  1. Simply polling my database every 5 or so seconds for changes. (This puts too much strain on my server since the Arduino would theoretically be running 24/7 and that would be about 17280).
  2. Use Webduino. (I’m having a hard time finding good documentation so if someone can point me in the right direction with this library then I would be happy to use it).
  3. Use another Arduino that is a dedicated server and will handle all the outputs while the rest of the Arduinos are clients.

p.s.
I think I have read almost every post regarding this topic and none of them have been able to solve this issue. Almost ALL the posts assume you just want to control the Arduino from the web and not the other way around in the same program.

Is it possible to truly have a server and client rotating on the Arduino without the server stopping the loop until it gets something?

yes, combined client/server test code below:

//zoomkat 7-03-12, combined client and server
//simple button GET with iframe code
//for use with IDE 1.0
//open serial monitor and send an g to test client and
//see what the arduino client/server receives
//web page buttons make pin 5 high/low
//use the ' in html instead of " to prevent having to escape the "
//address will look like http://192.168.1.102:84 when submited
//for use with W5100 based ethernet shields
//note that the below bug fix may be required
// http://code.google.com/p/arduino/issues/detail?id=605

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = {0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //assign arduino mac address
byte ip[] = {192, 168, 1, 102 }; // ip in lan assigned to arduino
byte gateway[] = {192, 168, 1, 1 }; // internet access via router
byte subnet[] = {255, 255, 255, 0 }; //subnet mask
EthernetServer server(84); //server port arduino server will use
EthernetClient client;
char serverName[] = "web.comporium.net"; // (DNS) zoomkat's test web page server
//byte serverName[] = { 208, 104, 2, 86 }; // (IP) zoomkat web page server IP address

String readString; //used by server to capture GET request 

//////////////////////

void setup(){

  pinMode(5, OUTPUT); //pin selected to control
  pinMode(6, OUTPUT); //pin selected to control
  pinMode(7, OUTPUT); //pin selected to control
  pinMode(8, OUTPUT); //pin selected to control

  //pinMode(5, OUTPUT); //pin 5 selected to control
  Ethernet.begin(mac,ip,gateway,gateway,subnet); 
  server.begin();
  Serial.begin(9600); 
  Serial.println("server/client 1.0 test 7/03/12"); // keep track of what is loaded
  Serial.println("Send an g in serial monitor to test client"); // what to do to test client
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) 
  {
    byte inChar;
    inChar = Serial.read();
    if(inChar == 'g')
    {
      sendGET(); // call client sendGET function
    }
  }  

  EthernetClient client = server.available();
  if (client) {
    while (client.connected()) {
      if (client.available()) {
        char c = client.read();

        //read char by char HTTP request
        if (readString.length() < 100) {

          //store characters to string 
          readString += c; 
          //Serial.print(c);
        } 

        //if HTTP request has ended
        if (c == '\n') {

          ///////////////
          Serial.print(readString); //print to serial monitor for debuging 

            //now output HTML data header
          if(readString.indexOf('?') >=0) { //don't send new page
            client.println(F("HTTP/1.1 204 Zoomkat"));
            client.println();
            client.println();  
          }
          else {   
            client.println(F("HTTP/1.1 200 OK")); //send new page on browser request
            client.println(F("Content-Type: text/html"));
            client.println();

            client.println(F("<HTML>"));
            client.println(F("<HEAD>"));
            client.println(F("<TITLE>Arduino GET test page</TITLE>"));
            client.println(F("</HEAD>"));
            client.println(F("<BODY>"));

            client.println(F("<H1>Zoomkat's simple Arduino 1.0 button</H1>"));

            // DIY buttons
            client.println(F("Pin5"));
            client.println(F("<a href=/?on2 target=inlineframe>ON</a>")); 
            client.println(F("<a href=/?off3 target=inlineframe>OFF</a>

")); 

            client.println(F("Pin6"));
            client.println(F("<a href=/?on4 target=inlineframe>ON</a>")); 
            client.println(F("<a href=/?off5 target=inlineframe>OFF</a>

")); 

            client.println(F("Pin7"));
            client.println(F("<a href=/?on6 target=inlineframe>ON</a>")); 
            client.println(F("<a href=/?off7 target=inlineframe>OFF</a>

")); 

            client.println(F("Pin8"));
            client.println(F("<a href=/?on8 target=inlineframe>ON</a>")); 
            client.println(F("<a href=/?off9 target=inlineframe>OFF</a>

")); 

            client.println(F("Pins"));
            client.println(F("&nbsp;<a href=/?off2468 target=inlineframe>ALL ON</a>")); 
            client.println(F("&nbsp;<a href=/?off3579 target=inlineframe>ALL OFF</a>")); 

            client.println(F("<IFRAME name=inlineframe style='display:none'>"));          
            client.println(F("</IFRAME>"));

            client.println(F("</BODY>"));
            client.println(F("</HTML>"));
          }

          delay(1);
          //stopping client
          client.stop();

          ///////////////////// control arduino pin
          if(readString.indexOf('2') >0)//checks for 2
          {
            digitalWrite(5, HIGH);    // set pin 5 high
            Serial.println("Led 5 On");
            Serial.println();
          }
          if(readString.indexOf('3') >0)//checks for 3
          {
            digitalWrite(5, LOW);    // set pin 5 low
            Serial.println("Led 5 Off");
            Serial.println();
          }
          if(readString.indexOf('4') >0)//checks for 4
          {
            digitalWrite(6, HIGH);    // set pin 6 high
            Serial.println("Led 6 On");
            Serial.println();
          }
          if(readString.indexOf('5') >0)//checks for 5
          {
            digitalWrite(6, LOW);    // set pin 6 low
            Serial.println("Led 6 Off");
            Serial.println();
          }
          if(readString.indexOf('6') >0)//checks for 6
          {
            digitalWrite(7, HIGH);    // set pin 7 high
            Serial.println("Led 7 On");
            Serial.println();
          }
          if(readString.indexOf('7') >0)//checks for 7
          {
            digitalWrite(7, LOW);    // set pin 7 low
            Serial.println("Led 7 Off");
            Serial.println();
          }     
          if(readString.indexOf('8') >0)//checks for 8
          {
            digitalWrite(8, HIGH);    // set pin 8 high
            Serial.println("Led 8 On");
            Serial.println();
          }
          if(readString.indexOf('9') >0)//checks for 9
          {
            digitalWrite(8, LOW);    // set pin 8 low
            Serial.println("Led 8 Off");
            Serial.println();
          }         

          //clearing string for next read
          readString="";

        }
      }
    }
  }
} 

//////////////////////////
void sendGET() //client function to send and receive GET data from external server.
{
  if (client.connect(serverName, 80)) {
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.0");
    client.println();
  } 
  else {
    Serial.println("connection failed");
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read();
    Serial.print(c);
  }

  Serial.println();
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop();

}

I actually did use that code as a basis for my testing but as I mentioned above, the loop doesn't work as it should because once it gets to EthernetClient client = server.available(); the loop stops until it receives a connection from the server. This means that if someone doesn't click a button on the website then the program will be stuck at that point forever and as a result won't be able to process any physical inputs like buttons being pushed, etc.

That's what I've been trying to figure out.

chessoriginal: I actually did use that code as a basis for my testing but as I mentioned above, the loop doesn't work as it should because once it gets to EthernetClient client = server.available(); the loop stops until it receives a connection from the server. This means that if someone doesn't click a button on the website then the program will be stuck at that point forever and as a result won't be able to process any physical inputs like buttons being pushed, etc.

That's what I've been trying to figure out.

That's not how it's supposed to work, i.e. in a non-blocking mode. Unless you reached the maximum number of sockets, which is just 4 for all clients ('true' clients and clients returned by server.available()). Without seeing the sendPOST() function one cannot say whether there may be some client hanging around. What I can see however is that client.stop() is only called if there is no client :~.

I retested the code I posted by uploading the code to the arduino, obtained the web page in IE, tested the page buttons and verified control of arduino pin 5 using a multimeter and output to the serial monitor. I tested the GET function using the serial monitor and the test page from my web server downloaded as expected. As a backup test I then used the reset button on the arduino to reset the arduino, then tested the buttons from the previous IE page and the control of pin 5 was as verified using a multimeter. I also reset the arduino by opening the serial monitor and sent a g to the arduino. The arduino downloaded the test page from my web server and sent it to the serial monitor. Perhaps your network setup doesn’t support the DNS action used in the GET function.

Yes, your code is correct, I realized what the issue is. I am using Yaler (https://yaler.net/) to connect to my Arduino from the internet (I’m under a pretty intense firewall) and I think that is locking up the connection somehow…which is unfortunate because I have been trying to figure this out for days. For anyone with experience using Yaler, is this normal behavior?

Thanks by the way, zoomkat.

The “s” on the end of “https” could be an issue for the arduino.

The link to my Yaler service doesn't have an 's' - it's http://xxxxxx.yaler.net/. I'm trying to try different configurations but so far I have confirmed that Yaler is what is holding up the connection.

chessoriginal: The link to my Yaler service doesn't have an 's' - it's http://xxxxxx.yaler.net/. I'm trying to try different configurations but so far I have confirmed that Yaler is what is holding up the connection.

Well, you did post the below:

I am using Yaler (https://yaler.net/) to connect to my Arduino from the internet (I'm under a pretty intense firewall) and I think that is locking up the connection somehow..

I was just parenthetically linking to the homepage of the service.

@chessoriginal (sorry for the late reply) You're right, the current implementation of the YalerEthernetServer library is blocking, while EthernetServer is non-blocking. I fear there's no way to change this, as YalerEthernetServer depends on an open (client) connection to the relay. Kind regards, Thomas (founder of Yaler.net)