reading data returned from HTTP GET in Arduino with Xbee wifi

Projects Programming
1

I started using Arduino about a month back. Very amateur in this regard. So i apologize first for any stupid question.

My project requires to read and write data from a mysql server. For writing to server I used post method and it works fine.
My wifi device is Xbee wifi. Phant library provided by sparkfun is being used here. I am using softwarewerial. So while reading, my code is like this:

softwareserial xb(6,7);
xB.print(phant.get());

phant.get is defined like this :

String Phant::get() {

  String result = "GET / .csv HTTP/1.1\n";
  result += "Host: " + _host + "\n";
  result += "Connection: close\n";

  return result;

}

My php code is supposed to return a series of numbers in comma separated value(csv) format. I have already checked this php file through other method (eg. postman ) and it does return the data in specified format.

The xbee wifi device is also 'hitting' (if thats the term !) the server, because if I put a wrong syntax in it, I can see the error reporting in the php error log file. So connection is alright.

my question is here, xb.print() is by default supposed to return an int value. where as the get function inside it is returning a string value. how can i retrieve the returned value by phant.get() ?

I will appreciate any answer. thanks.

2

to add, even when i do a serial read after calling phant.get(), i can't get any data.

int response; 
    xB.print(phant.get());
    
    while(xB.available())
    {
    response=xB.read();
    Serial.print(response);
    Serial.print(',');
    //Serial.print('\n');
    }

This is returning me with some data Not from the database. it remains same when i empty the database.

3

Records in a GET request are separated by carriage return/line feed, not just one.

Do you really have a file on the server called .csv? A .csv file is NOT executable, so making a GET request is NOT going to do anything, and will NOT generate a return stream to read.

4

here is my php right now :

<?php
echo "Hello World";
?>

this is saved as index.php and I do get the response "hello world" when i type in my browser 'localhost'.

my get function :

String Phant::get() {

  String result = "GET / HTTP/1.1\n";
  result += "Host: " + _host + "\n";
  result += "Connection: close\n";

  return result;

}

still no luck :frowning:

5

So, you have a file called index.php. Why are you not GETting index.php? You realize, I hope, that the browser does some behind-the-scenes magic based on the URL you are asking for and what is actually on the server. Your Arduino is not doing that magic. You should not make assumptions.

6

Well, this format worked with the "POST" routine. I didn't have to explicitly mention index.php.

However, I managed to get some response now. I had to introduce a delay of one second between the GET request and serial Read().

however, it says Bad Request. I now have to deal with that.

I don't have much idea about how to discard the header and extract the data. But I found some reference to work on. Learning every moment :slight_smile:

Thanks PaulS for the reply. :slight_smile:

7

I had to introduce a delay of one second between the GET request and serial Read().

This doesn't make sense. The reply to the GET request does not come through the serial port.

You should not use a delay() to wait for a reply. You make a request. On some pass through loop(), some of the reply will be ready to read.

8

what do you mean when u said the response of GED doesn't come through serial port.. my xbee wifi is connected using softwareserial. and i am receiving data by using for example xB.read(). i am getting confused.

9

If you are using a WiFi device that is connected to Serial, then forget what I said about the response not coming through Serial. The Ethernet shield and WiFi shield do not use Serial to communicate with the Arduino.

10

Oh wow. that helps. I will quit trying then .

the reason why i was trying to use serial port is because with software serial, some of my data packets were missing while receiving. any suggestion in that regard?

11

I will quit trying then

Why?

any suggestion in that regard?

The usual. Post a few random snippets of various sketches and lead us to believe that they are all from the same sketch.

No, wait. That's supposed to be "Post ALL of your code!"

12

I meant, i will quit trying to use the xbee wifi device with the serial port.

My codes have so many comments and flags and ugly right now. will try to clear it up and post here later.

13

here is the code part where I am trying to, as of now, just show the data received.

void loop()
{
  if (millis() > (lastUpdate + UPDATE_RATE))
  {
    xB.print(phant.get());
    Serial.print("\n");
    Serial.print(phant.get()); // GET request to the server.
   
    Serial.print("...................\n\n\n\n");
    lastUpdate=millis();
  
    String Response="";
   
   
    while(xB.available())
         {
          //Response.concat(char(xB.read()));
           c=char(xB.read());
           delay(1);
          Serial.print(c);
         }
     //Serial.print(Response);
  }
}

I am expecting an array of 7 numbers afther the header. I am receiving these data :

HTTP/1.1 200 OK
Date: Wed, 02 Jul 2014 15:53:51 GMT
Server: AHTTP/1.1 200 OK
Date: Wed, 02 Jul 2014 15:54:03 GMT
Server: Apache/2.4.9 (Win32) OpenSSL/1.0.gPP551
XPrdB:PP551
CnetLnt:3
Cneto:coeCnetTp:tx/tl

rf ,6,0 4 ,2,0

the array part at the end is not what i have on the server. Only a few letters are coming through.

14

the array part at the end is not what i have on the server. Only a few letters are coming through.

Perhaps you should post the array that is on the server so it can be compared to what is received. Perhaps you have a server issue.

15

If I run the php from my browser, this is the data I get :

Arif, 0, 61, 0, 34, 0, 21, 0,

What kind of server issue i can have? any hint?

16

I can receive the same data on another machine using the same wifi network.

17

arif_1:
I can receive the same data on another machine using the same wifi network.

Could be an issue with your arduino wifi setup. As WIFI is now a some what generic wireless term, does the xbee directly connect to your router? Below is a link to a simple test file link you might try to see if it is correctly received.

http://web.comporium.net/~shb/arduino.txt

18

Update :

I feel, the Xbee module has some sort of buffer size limit ( although data sheet says its 2083 bytes, and my total packet is 309 bytes)

Here is the data sent from the server ( got from wireshark)

HTTP/1.1 200 OK
Date: Wed, 02 Jul 2014 20:36:29 GMT
Server: Apache/2.4.9 (Win32) OpenSSL/1.0.1g PHP/5.5.11
X-Powered-By: PHP/5.5.11
Content-Length: 31
Connection: close
Content-Type: text/html
Arif, 0, 61, 0, 34, 0, 21, 0,

However, using a little delay after GET request, I consistently am receiving a part of it only :

HTTP/1.1 200 OK
Date: Thu, 03 Jul 2014 02:38:56 GMT
Server: A

This is consistent and I am getting this everytime (with change in timestamp).

Here is the code segment :

void loop()
{
  if (millis() > (lastUpdate + UPDATE_RATE))
  {
    xB.print(phant.get());
    delay(3000);
    lastUpdate=millis();
  
   while(xB.available())
         {
           c=char(xB.read());
           delay(1);
          Serial.print(c);
         }
     }
}

I dont know if there is a way to limit the packet size from the server end. I will search on it. Any input will help, again.

19

However, using a little delay after GET request, I consistently am receiving a part of it only :

Get ri of the delay(), and the serial buffer will quite overflowing.

20

After removing delays(), its better : I have no idea why. Now i can read the header completely. however, the body is missing or dropping data

my php is :

<?php
echo "hello world \n\n hello world \n\n hello world \n\n hello world \n\n hello world \n\n hello world";
?>

and I can read from serial monitor this :

HTTP/1.1 200 OK
Date: Fri, 04 Jul 2014 05:23:36 GMT
Server: Apache/2.4.9 (Win32) OpenSSL/1.0.1g PHP/5.5.11
X-Powered-By: PHP/5.5.11
Content-Length: 86
Connection: close
Content-Type: text/html

hll wold

helo ord 
 elo wrl 
 helowold 
 elo orl 
 hllowold

the "hello world" is missing different characters at different request.

( i am sort of updating my experience as a new bie here, not sure if its appropriate. I appreciate the help).