HTTP errors requesting twitter server...

Projects Networking, Protocols, and Devices
1

hello,
i'm working 2 weeks with the ARduino ethernet and Arduino 1.0 IDE.
I'm experimenting some problems using HTTP connections.

It is very extrange, because this problems depends of the place i connect to internet. I mean the internet connection. I try in 3 diferents connections:

Fiber optic very good connection: No problems!
ADSL 10Mb normal connection: 15% errors
3G connection: 25% errors

I started to work with TwitterClient and in the begining i has a lot of problems with two much connections. After 5 connection client.connect(serverName, 80) not work, so i think we are in the limit of socket (4). I resolve this put another client.stop() and client.flush() after to call connectToserver() .

After this i experiment a lot of error in connections...
is like client.Available() not receive nothing!
Is there any timer or something? because this works very good in fiber optic connection to internet.
Then I suspect that might be the server's response time... some times works good and some times works very bad.

What do you think?
Any ideas?

This is cophy of terminal debug. You can see the are some connetions without request.

connecting to server...
making HTTP request...
>@RoxyHorror meet @tigoe you guys share the love for rollerderby :)
connecting to server...
making HTTP request...
connecting to server...
making HTTP request...
>@RoxyHorror meet @tigoe you guys share the love for rollerderby :)
connecting to server...
making HTTP request...
>@RoxyHorror meet @tigoe you guys share the love for rollerderby :)
connecting to server...
making HTTP request...
connecting to server...
making HTTP request...
>@RoxyHorror meet @tigoe you guys share the love for rollerderby :)

And this is the firmware with the small modification:

/*
  Twitter Client with Strings
 
 This sketch connects to Twitter using an Ethernet shield. It parses the XML
 returned, and looks for <text>this is a tweet</text>
 
 You can use the Arduino Ethernet shield, or the Adafruit Ethernet shield, 
 either one will work, as long as it's got a Wiznet Ethernet module on board.
 
 This example uses the DHCP routines in the Ethernet library which is part of the 
 Arduino core from version 1.0 beta 1
 
 This example uses the String library, which is part of the Arduino core from
 version 0019.  
 
 Circuit:
  * Ethernet shield attached to pins 10, 11, 12, 13
 
 created 21 May 2011
 by Tom Igoe
 
 This code is in the public domain.
 
 */
#include <SPI.h>
#include <Ethernet.h>


// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = { 
  0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x01 };
IPAddress ip(192,168,1,20);

// initialize the library instance:
EthernetClient client;

const int requestInterval = 20000;  // delay between requests

char serverName[] = "api.twitter.com";  // twitter URL

boolean requested;                   // whether you've made a request since connecting
long lastAttemptTime = 0;            // last time you connected to the server, in milliseconds

String currentLine = "";            // string to hold the text from server
String tweet = "";                  // string to hold the tweet
boolean readingTweet = false;       // if you're currently reading the tweet

void setup() {
  // reserve space for the strings:
  currentLine.reserve(256);
  tweet.reserve(150);

// initialize serial:
  Serial.begin(9600);
  // attempt a DHCP connection:
  if (!Ethernet.begin(mac)) {
    // if DHCP fails, start with a hard-coded address:
    Ethernet.begin(mac, ip);
  }
  // connect to Twitter:
  connectToServer();
}



void loop()
{
  if (client.connected()) {
    if (client.available()) {
      // read incoming bytes:
      char inChar = client.read();

      // add incoming byte to end of line:
      currentLine += inChar; 

      // if you get a newline, clear the line:
      if (inChar == '\n') {
        currentLine = "";
      } 
      // if the current line ends with <text>, it will
      // be followed by the tweet:
      if ( currentLine.endsWith("<text>")) {
        // tweet is beginning. Clear the tweet string:
        readingTweet = true; 
        tweet = "";
      }
      // if you're currently reading the bytes of a tweet,
      // add them to the tweet String:
      if (readingTweet) {
        if (inChar != '<') {
          tweet += inChar;
        } 
        else {
          // if you got a "<" character,
          // you've reached the end of the tweet:
          readingTweet = false;
          Serial.println(tweet);   
          // close the connection to the server:
          client.stop(); 
        }
      }
    }   
  }
  else if (millis() - lastAttemptTime > requestInterval) {
    // if you're not connected, and two minutes have passed since
    // your last connection, then attempt to connect again:
    client.flush();
    client.stop();
    connectToServer();
  }
}

void connectToServer() {
  // attempt to connect, and wait a millisecond:
  Serial.println("connecting to server...");
  if (client.connect(serverName, 80)) {
    Serial.println("making HTTP request...");
  // make HTTP GET request to twitter:
    client.println("GET /1/statuses/user_timeline.xml?user=arduino&count=1 HTTP/1.1");
    client.println("HOST: api.twitter.com");
    client.println();
  }
  // note the time of this connect attempt:
  lastAttemptTime = millis();
}
2

You should keep waiting for input until the twitter server closes the connection. That is the signal it is finished, not the "" message.

If you close the connection with data in the rx buffer, the connection will stay open. You need to read the rx buffer until the server closes the connection. Here is a code snippet that should empty the rx buffer until the server closes the connection.

while(client.connected())
{
   while(client.available())
   {
      inChar = client.read();
      // here is where you want to do your search for the "<text>" message
   }
}

// now close your end!
client.stop();
3

ok,
i will try this night and come back with news!

thanks

4

hello again,
i try this option and while is very dangerous...
some times freeze in the while... and the server never disconnect... i put a timer to break in this case, not work, wait in connecting...
but this should not happend. Exactly happens always is in the connection number 6!!!

  while (client.connected()) //while
  {
    while (client.available()) //while
    {
      char c = client.read();
      Serial.print(c);

      lastConnected = 0;
      
      if(millis() - lastConnectionTime > postingInterval)
      {
        client.flush();
        client.stop(); 
      }
      
    }
    
  }
  
  if((!client.connected()) && !lastConnected)
  {
    Serial.println("Disconnecting...");
    lastConnected = 1;
    client.stop();
    connectToserver();
  }
5

If you use HTTP/1.1 then you must have a "Connection: close" header if you want the server to close the socket, otherwise it keeps it open. HTTP/1.0 might work (but I think that means losing the "Host:" header. RFC2616 for more details.

6

ok thanks,
better with HTTP/1.0

the problem was that sometimes the server close the connection without receive nothing...
i have to detect this state and resend again a new connection to server.

But this errors depends a lot of the speed of connection to internet. This is strange.
Fiber, ADSL or 3G has very diferent behaviors with the same hardware.

/A

7

Are you ever updating requestInterval or postingInterval?

8

Are you ever updating requestInterval or postingInterval?

yes, i changed to 25000 because is the rate limit of twitter. It only permit 150 msg every hour.
The original example has 60 seconds, but this is too slow for me. I need an application for real time or near to real time.

Last version,
i implement an if to detect if the msg was received or not to do a send a new request in case client.available not receive.

while (client.connected()) 
  {
    while (client.available()) 
    {
 
      //char c = client.read();
      //Serial.print(c);

      lastConnected = 0;

      if(!tweetReady)
      {
        
        Serial.println("request received");
        
        // Finder function
        if(finder.find("<status>"))
        { 
          if(finder.getString("<text>","</text>",tweet,140) != 0)
          {
            Serial.print("Tweet: ");
            Serial.println(tweet);   
            tweetReady = 1;    
          }
        } 
        

      }  
      else
        char c = client.read();
      
    }

  }

// DISCONNECT FUNCTION

 
  if( ((!client.connected()) && !lastConnected) ) 
  {
    Serial.println("Disconnecting by server");
    lastConnected = 1;
    client.flush();
    client.stop();
  }
  else
  {
     Serial.print('.'); 
     delay(1000);
  }
  
  
  if ( (!client.connected() ) && (millis() - lastConnectionTime > postingInterval) )
  {
    Serial.println();
    Serial.println("Disconnecting by timer");
    lastConnected = 1;
    client.flush();
    client.stop();
    connectToserver();
  }



  if((!client.connected()) && !tweetReady) // client.available close connection without data - ERROR!!!
  {
     Serial.println("Disconnecting without request");
     client.flush();
     client.stop();
     connectToserver();
  }


}