Again search.twitter

Projects Networking, Protocols, and Devices
1

My old thread http://arduino.cc/forum/index.php/topic,45921.0.html is locked so ...
I hate to give up especially if I know that there must be a solution and I am only too dump to find it. So let's try again some crowdsourcing:

I want to connect to twitter and only search a tweet. (I have hardcoded the IP. I found other solutions with an external server which I do not like. So I started again to let Arduino do the work:

#include <Ethernet.h>

byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192,168,2,177 };
byte gateway[] = { 192, 168, 2, 1};
byte server[] = { 128, 242, 240, 148 }; // search twitter

#define LOGGING
Client client(server, 80);

#ifdef LOGGING
void printIPAddress(byte* aAddress)
{
  Serial.print((int)aAddress[0]);
  Serial.print(".");
  Serial.print((int)aAddress[1]);
  Serial.print(".");
  Serial.print((int)aAddress[2]);
  Serial.print(".");
  Serial.print((int)aAddress[3]);
}

void printlnIPAddress(byte* aAddress)
{
  printIPAddress(aAddress);
  Serial.println();
}
#endif

void setup() {
  Ethernet.begin(mac, ip, gateway);
  Serial.begin(9600);
  delay(1000);
  Serial.println("connecting...");

  if (client.connect()) {
    Serial.println("connected");
    Serial.print("Using Server: ");
    printlnIPAddress(server);
    
    // Make a HTTP request:
    client.println("GET /search.atom?q=Morse&rpp=1 HTTP/1.1 ");
    client.println("Host: search.twitter.com");
    client.println("Accept: text/xml;q=0.9");
    client.println();
  } 
  else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    Serial.print(c);
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();

    // do nothing forevermore:
    for(;;)
      ;
  }
}

The code is strictly from the book. Here is the crucial part:

** client.println("GET /search.atom?q=Morse&rpp=1 HTTP/1.1 ");**
** client.println("Host: search.twitter.com");**
** client.println("Accept: text/xml;q=0.9");**
** client.println();**

I found out, that you have to define HTTP/1.1, for HTTP/1.0 does not work anymore. Twitter says that the query must be URL-encoded. But the browser (Firefox, Chrome) and with php the GET-command works fine.

Still I get the message from twitter:

connected
Using Server: 128.242.240.148
HTTP/1.1 406 Not Acceptable
Date: Fri, 18 Feb 2011 22:11:11 GMT
Server: hi
Status: 406 Not Acceptable
X-Transaction: 1298067070-49239-28953
X-RateLimit-Limit: 150
Last-Modified: Fri, 18 Feb 2011 22:11:10 GMT
X-RateLimit-Remaining: 131
X-Runtime: 0.08282
X-Transaction-Mask: 0b5b266a28469a7b52ded76c9a66f018
Content-Type: text/html; charset=utf-8
Content-Length: 1
Pragma: no-cache
X-RateLimit-Class: api
X-Revision: DEV
Expires: Tue, 31 Mar 1981 05:00:00 GMT
Cache-Control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0
X-RateLimit-Reset: 1298067366
Set-Cookie: k=87.158.191.193.1298067070980993; path=/; expires=Fri, 25-Feb-11 22:11:10 GMT; domain=.twitter.com
Set-Cookie: guest_id=129806707098362725; path=/; expires=Sun, 20 Mar 2011 22:11:10 GMT
Set-Cookie: _twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCAig0zouAToHaWQiJTFkZWZhYmFkYWE0OTY0%250ANjEyMjhiN2Y5YjU4ZGY4ZTFlIgpmbGFzaElDOidBY3Rpb25Db250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNoewAGOgpAdXNlZHsA--9a327e01e2a258cf186936f58b9fd70feb63bf79; domain=.twitter.com; path=/; HttpOnly
Vary: Accept-Encoding
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Connection: close

Any idea what is still wrong in my query? Or how do I code an URL-encoded query?

There is still hope, that someone may find the answer :stuck_out_tongue_closed_eyes:

Thanks for your patience
Hajo

2

Have you done any research on what a 406 error code means?
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

10.4.7 406 Not Acceptable

The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.

Unless it was a HEAD request, the response SHOULD include an entity containing a list of available entity characteristics and location(s) from which the user or user agent can choose the one most appropriate. The entity format is specified by the media type given in the Content-Type header field. Depending upon the format and the capabilities of the user agent, selection of the most appropriate choice MAY be performed automatically. However, this specification does not define any standard for such automatic selection.

Note: HTTP/1.1 servers are allowed to return responses which are
not acceptable according to the accept headers sent in the
request. In some cases, this may even be preferable to sending a
406 response. User agents are encouraged to inspect the headers of
an incoming response to determine if it is acceptable.

If the response could be unacceptable, a user agent SHOULD temporarily stop receipt of more data and query the user for a decision on further actions.

You should look at what PHP actually generates and sends in response to the GET command.

3

Hello Paul,

thanks for the hint. Yes I had a look to the 406-error but couldn't interpret it into correct coding. I had a look at Understanding Browser HTTP Accept Headers: Firefox, Internet Explorer, Opera, and WebKit (Safari / Chrome) and added the additional lines, but without success.

The problem with the php-code is, that it runs on a apache server and the answer is interpreted by a browser and I am no capable to see what the browser does. I thought about using a webspace with php as proxy and an own IP, but that is overload 8)

Thanks
Hajo

4

PHP is open source. You can always look at the source code to see how it handles a GET command.

5

Problem solved! I asked in the Twitter Dev group and Nikita provided within minutes the answer: I had to connect to another IP: 199.16.156.11.

Thanks and have a nice weekend.

Hajo