[SOlved]GET/ requests Headache

I'm just wanting to extract some email information from my IMAP.google.com.

My arduino code:

#include <SPI.h>
#include <Ethernet.h>

int relayPin = 9;
char Str[11];
int prevNum = 0;
int num = 0;
long onUntil = 0;

long pollingInterval = 5000; // in milliseconds
long onTime = 60000; // time, in milliseconds, for lights to be on after new email or comment

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = {  0x90, 0xA2, 0xDA, 0x0D, 0x78, 0xE1 };
char serverName[] = "winacro.com";



// Initialize the Ethernet client library
// with the IP address and port of the server 
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
  Serial.begin(9600);
  pinMode(relayPin, OUTPUT);
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }
  // give the Ethernet shield time to initialize:
  delay(2000);

}

void loop()
{
    Serial.println("connecting...");

  // if you get a connection, report back via serial:
  
  if (client.connect(serverName, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET /PHP/email.php HTTP/1.1");
    client.println("HOST: winacro.com");
    client.println();
    int timer = millis();
    //delay(1000);
  } 
  else {
    // if you didn't get a connection to the server:
    Serial.println("connection failed");
    
  }
	// if there's data ready to be read:
  if (client.available()) {
     int i = 0;
     
     //put the data in the array:
     do {
       Str[i] = client.read();
       i++;
       delay(1);
     } while (client.available());
     
     // Pop on the null terminator:
     //Str[i] = '\0';
     //convert server's repsonse to a int so we can evaluate it
     Serial.print(Str);
     num = atoi(Str); 
     
     Serial.print("Server's response: ");
     Serial.println(num);
     Serial.print("Previous response: ");
     Serial.println(prevNum);
     if (prevNum < 1)
     { //the first time around, set the previous count to the current count
      prevNum = num; 
      Serial.println("First comment count stored.");
     }
     if (prevNum > num)
     { // handle if count goes down for some reason
      prevNum = num; 
     }
  }
  else
    {
     Serial.println("No response from server."); 
    }
    Serial.println("Disconnecting.");
    client.stop();
    if(num > prevNum) {
      Serial.println("New Comment.");
      digitalWrite(relayPin, HIGH);
      prevNum = num;
      onUntil = millis() + onTime;
    }
    else if(millis() > onUntil)
    {
     digitalWrite(relayPin, LOW); 
    }
  //delay(pollingInterval);
}

This is what I get:

connecting...
connected
No response from server.
Disconnecting.
<?php
/* connect to gmail */
$hostname = "{imap.gmail.com:993/imap/ssl}INBOX";
$username = "xcxc@winwin.com";
$password = "1123412";

/* try to connect */
$inbox = imap_open($hostname,$username ,$password) or die('Cannot connect to Gmail: ' . imap_last_error());

$status = imap_status($mbox, "{imap.gmail.com:993/imap/ssl}INBOX", SA_MESSAGES);
if ($status) {
  echo $status->messages;
}
?>

Does the server access or error log have anything interesting to say?

-br

You are not waiting for the server to respond. It has things to do before it can send you a response, and your code is not waiting. If the server dos not respond immediately, your code will always return "no response from server". Here is a sketch that waits for the server response and closes the connection correctly.
http://playground.arduino.cc/Code/WebClient

This will cause serious problems.

     //put the data in the array:
     do {
       Str[i] = client.read();
       i++;
       delay(1);
     } while (client.available());

Use something like this to keep from overflowing the Str array.

     //put the data in the array:
     do {
       if(i < 10) {
         Str[i] = client.read();
         i++;
         delay(1);
       }
     } while (client.available());

Use something like this to keep from overflowing the Str array.

But not exactly that. The do/while body will always be executed as least once, performing a read() even if there is nothing to read.

PaulS:

Use something like this to keep from overflowing the Str array.

But not exactly that. The do/while body will always be executed as least once, performing a read() even if there is nothing to read.

Thanks, Paul. I didn't notice that. I wouldn't use this anyway because it has no reference to client.connected() or a timeout. The playground code does. I was only concerned about overflowing that array.

NI$HANT,

How are you doing? I just started tinkering with wifi shield, which is pretty similar to Ethernet shield. It seems that you didn't wait for server response like others already pointed out.

Could you give a pointer on where to learn syntax like below?

    client.println("GET /PHP/email.php HTTP/1.1");
    client.println("HOST: winacro.com");
    client.println();

I wanted to be able to send emails BTW.

I wanted to be able to send emails BTW.

That I can answer.
http://playground.arduino.cc//Code/WiFiEmailClient
This code needs some improvement to the wifi shield setup. It has no error checking, and that could cause problems for a novice user.

Thanks SurferTim!

Liudr Sir,

I'm doing great I hope you are fine too!

I have good experience with WiFi so I would say the UART based wiFi modules from Roving networks are a LOVE affair I recently wrote a Android App doing TCP/IP and all I needed to handle in Embedded side of system is UART comm of the wifi chip with that of the ATmega, great little thing.
Which WIfi chip is there on your Shield?

@SurferTIM-> I would just try delaying a few seconds after that part you pointed out in the same code I have and then come back to you!

NI$HANT:
@SurferTIM-> I would just try delaying a few seconds after that part you pointed out in the same code I have and then come back to you!

Only if you don't want to take the time to do it right. But then you already have a headache, so what will it matter? :slight_smile:

edit: Here is the right way, in the event you get tired of that headache.

  while(client.connected())  {
    while(client.available())    {
      inChar = client.read();
      Serial.write(inChar);
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
    }

    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }

  Serial.println();

  Serial.println(F("disconnecting."));
  // close client end
  client.stop();

Only if you don't want to take the time to do it right. But then you already have a headache, so what will it matter?

Nah! man.. thanks for keep coming back

SHIT! now my webserver went down!

You will have problems if you do not use the code I added above. It has all the stuff that will make your headache go away. Nothing in that code is a waste of memory. It all has a purpose.

The "while(client.available())" loop only reads one packet at a time. So if you do not stay in the "while(client.connected())" loop, you only read the first packet.

The theory is the server will send packets, and when it is finished, it will close the connection. Then you as the client close your end. If this was "perfect world", that would be all you need.

However, this is not "perfect world", this is "real world". Stuff fails here, and if the connection breaks (fails) while waiting for the server response, it will stay in the "while(client.connected())" loop forever. To prevent that, there is a timeout routine that will close the connection if it does not receive a packet for 10 seconds. That is what loopCount does.

You can avoid all the headaches it caused me to get those problems out, or you can figure it out yourself like I did.

edit: Your web server is in "real world". If it failed while the Arduino was waiting for the response, and you had no timeout, that would have been a fail on the Arduino also.

edit: Your web server is in "real world". If it failed while the Arduino was waiting for the response, and you had no timeout, that would have been a fail on the Arduino also.

I'm also running my website side by side to see if its caused by my webserver fail or my Arduino problem (while my code is certainly having problems as you pointed out) but right now anything isn't working because my 8 Dollar per year web space is down!

ANOTHER thing is that Im going to remove the following snippet from my code and replace it with your Above recommendation!

	// if there's data ready to be read:
  if (client.available()) {
     int i = 0;
     
     //put the data in the array:
     do {
       Str[i] = client.read();
       i++;
       delay(1);
     } while (client.available());
     
     // Pop on the null terminator:
     //Str[i] = '\0';
     //convert server's repsonse to a int so we can evaluate it
     Serial.print(Str);
     num = atoi(Str); 
     delay(2000);
     Serial.print("Server's response: ");
     Serial.println(num);
     Serial.print("Previous response: ");
     Serial.println(prevNum);
     if (prevNum < 1)
     { //the first time around, set the previous count to the current count
      prevNum = num; 
      Serial.println("First comment count stored.");
     }
     if (prevNum > num)
     { // handle if count goes down for some reason
      prevNum = num; 
     }
  }
  else
    {
     Serial.println("No response from server."); 
    }

NI$HANT:
Liudr Sir,

I'm doing great I hope you are fine too!

I have good experience with WiFi so I would say the UART based wiFi modules from Roving networks are a LOVE affair I recently wrote a Android App doing TCP/IP and all I needed to handle in Embedded side of system is UART comm of the wifi chip with that of the ATmega, great little thing.
Which WIfi chip is there on your Shield?

Thanks NI$HANT! I am doing fine! I am using the official Arduino WiFi shield R3 for a project. This shield has a dedicated library comparable to the Ethernet library in terms of classes and methods. So you're saying that I can get away with a UART ("serial") module so skip the library? Will it then need to switch between text and command mode with some special string like "!!!" or "+++"? I just don't know enough about network shields and modules.

SurferTim,

Thank you for the detailed explanation here especially the real world situation. I will use your code in my project so it doesn't fall apart.

Sir Liudr,

I think the Arduino WiFi shield doesn't have that kind UART communication facility everything needs to work through its library!
The biggest thing working with a ARduino WiFi shield is its cost at the very first time I was turned down with it, Learning to work with something which will hardly have any end product place is not good , and in case of Arduino WiFi shield this is the case because of its cost, I that's why picked up a RN-171 and Now im eyeing that very cheap TIcc3000 chip that can go upto $10 for 1000 pcs that's pretty cheap when it comes to connecting anything to the Internet!

Thanks for your reply NI$HANT. I am working on a project that will be cost effective and potentially have more functionality if I used arduino wifi shield. I am more confident with Arduino team's ability to write a decent firmware and library than other makers also.

I am more confident with Arduino team's ability to write a decent firmware and library than other makers also.

RN-171 and other many flavors are from MicroChip and you know microchip is the best when it comes to good quantity manufacturing and thus cost gets dropped, neverthless this is my analysis.

@ SurferTIM , I added your code and its below(just commented out>> client.println("HOST: winacro.com"):wink:

/*
  
#include <SPI.h>
#include <Ethernet.h>

int relayPin = 9;
char Str[11];
int prevNum = 0;
int num = 0;
long onUntil = 0;
int connectLoop;

long pollingInterval = 5000; // in milliseconds
long onTime = 60000; // time, in milliseconds, for lights to be on after new email or comment

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = {  0x90, 0xA2, 0xDA, 0x0D, 0x78, 0xE1 };
char serverName[] = "winacro.com";



// Initialize the Ethernet client library
// with the IP address and port of the server 
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;

void setup() {
  Serial.begin(9600);
  pinMode(relayPin, OUTPUT);
  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }
  // give the Ethernet shield time to initialize:
  delay(2000);

}

void loop()
{
    Serial.println("connecting...");

  // if you get a connection, report back via serial:
  
  if (client.connect(serverName, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET /PHP/email.php HTTP/1.1");
    //client.println("HOST: winacro.com");
    client.println();
    int timer = millis();
    //delay(1000);
  } 
  else {
    // if you didn't get a connection to the server:
    Serial.println("connection failed");
    
  }
	// if there's data ready to be read:
  while(client.connected())  {
    while(client.available())    {
      char inChar = client.read();
      Serial.write(inChar);
      // set connectLoop to zero if a packet arrives
      connectLoop = 0;
    }

    connectLoop++;

    // if more than 10000 milliseconds since the last packet
    if(connectLoop > 10000)    {
      // then close the connection from this end.
      Serial.println();
      Serial.println(F("Timeout"));
      client.stop();
    }
    // this is a delay for the connectLoop timing
    delay(1);
  }

  Serial.println();

  Serial.println(F("disconnecting."));
  // close client end
  client.stop();   
  if(num > prevNum) {
      Serial.println("New Comment.");
      digitalWrite(relayPin, HIGH);
      prevNum = num;
      onUntil = millis() + onTime;
    }
    else if(millis() > onUntil)
    {
     digitalWrite(relayPin, LOW); 
    }
  //delay(pollingInterval);
}

and I get the following with it:

connecting...
connected
HTTP/1.1 400 Bad Request
Date: Thu, 18 Apr 2013 18:25:33 GMT
Server: Apache
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.

</p>
</body></html>

disconnecting.
connecting...

if i keep the client.println("HOST: winacro.com"); then i'm not able to connect!