How to parse an HTTP Get request, without all the HTTP dross

Using Arduino Programming Questions
#1

I’m pretty new to Arduino programming, and am trying to parse data from an HTTP Get request from a CC3000. I’ve found several other forum posts elsewhere with a similar subject, but none of them resolves my particular problem. Which is...

I connect to the server fine, it displays the results of the request fin the Serial window, and I can even capture the whole output as a string… But that’s the problem, I don’t know how to capture just the results of the request (which will be a two- or three-digit int) rather than capturing all the HTTP garbage that gets sent also. (such as "HTTP/1.1 200 OK Date: Tue, 07 Jan 2014 10:02:05 GMT Server: Apache”, etc. etc.)

If it helps, the PHP script on the server is here: http://faludidesign.com/TEMP/thermostat/record.php .
The Arduino sends it some variables in the following form: http://faludidesign.com/TEMP/thermostat/record.php?rf=700&af=705&ot=704&h=0&c=1&sh=22
And then the server responds with a number, which is a 2- or 3-digit int. Nothing else. You can see by going to the URL in a browser.

I’m currently getting each character as it comes from the server, and concatenating them into a string (which works), and then trying to read just the last three characters of that string and convert it to an int (which isn’t working). I’ve tried endsWith, toCharArray, strncopy, memcpy, getBytes, none of it works. I’ll get errors like ‘function not declared’, or ‘invalid conversion from ‘int’ to const char*’, or ‘invalid conversion from ‘const char’ to const char*’ or ‘no matching function call to String::toCharArray(String&, int)’, etc.

I’m flailing here. Please help!

If there’s a way to just get the few characters the PHP script on the server sends to the CC3000, that’d be ideal. If not, what’s the best way to get the last three characters of the string & make them an int?

Here’s this section of the code:

    while (www.connected()) {
            while (www.available()) {
              char c = www.read();  //Reads the output of the HTTP Get Request
              Serial.print(c);
              HTTPoutputString += c;  //concatenates each character onto the end of the string as it comes
              
              }
            }  
          www.close();
          
          String Buf;
          Buf = HTTPoutputString.substring(60,20);
          
          char Foo = HTTPoutputString.charAt(3);
                    
          // DIDN'T WORK:   Buf = HTTPoutputString.endswith(6);
          // DIDN'T WORK:   HTTPoutputString.toCharArray(Buf, 3);
          // DIDN'T WORK:   strncopy(HTTPoutputString, Buf, 3);
          // DIDN'T WORK:   HTTPoutputString.getBytes(Buf,1);
          // DOESNT'T WORK:  memcpy( HTTPoutputString, readString, strlen(readString)+1 );  ;
          //HTTPoutputString = readString; 
          //SetpointHeat = HTTPoutputString;
          Serial.print("\nHTTPoutputString: ");   Serial.print(HTTPoutputString);  Serial.print("\n");
          Serial.print("\n\nBuf: ");   Serial.print(Buf);  Serial.print("\n");
          Serial.print("\n\nFoo: ");   Serial.print(Foo);  Serial.print("\n");
          Serial.println(F("-------------------------------------"));
          HTTPoutputString = "";
          Buf = "";
          
          /* You need to make sure to clean up after yourself or the CC3000 can freak out */
          /* the next time your try to connect ... */
          Serial.println(F("\n\nDisconnecting"));
          cc3000.disconnect();
#2

But that’s the problem, I don’t know how to capture just the results of the request (which will be a two- or three-digit int) rather than capturing all the HTTP garbage that gets sent also. (such as "HTTP/1.1 200 OK Date: Tue, 07 Jan 2014 10:02:05 GMT Server: Apache”, etc. etc.)

The results of the request ARE all the "garbage". That only a part of the response is of interest to you is rather inconsequential.

But, that's why strtok() and other string handling functions were invented. YOU get to collect all the data (including the etc. etc. part that contains the information that you want) into a string, and then use strtok() to extract tokens. Fortunately, the response is delimited by carriage returns and line feed, among other things.

’m currently getting each character as it comes from the server, and concatenating them into a string

No, you aren't. HTTPoutputString is not a string. It's a String, which is a completely different beast.

          char Foo = HTTPoutputString.charAt(3);

So, Foo contains the 4th character from the response (a 'P', if the output you showed above is correct.

Once you have the useless String, use toCharArray() to convert it to a useable string, then use strtok() to extract the tokens. The last one will (probably) be the one that you want. Use itoa() to convert that token to an int.

#3

The HTTP headers are separated from the HTTP body by the character sequence

(0x0D 0x0A 0x0D 0x0A)
so you just need to read till you see this sequence, everything before it is HTTP header
lines (each separated by just one sequence), everything after it is the body
up to end-of-stream (EXCEPT if Connection: keep-alive has been specified IIRC, in which
case you need to parse the headers to find the body length count)

If the body has been encoded its up to you to handle that, hopefully its plain text.

#4

Thanks, MarkT, this is helpful. It sounds like you're implying that I could use the String.startsWith command in a syntax like this:

Buf = HTTPoutputString.startsWith("\r\n\r\n",3);

Right?

Unfortunately I keep getting the error: "invalid conversion from 'unsigned char' to 'const char*' ". I've tried searching other forums for that error message, and don't understand anything people are saying in them. I don't know how to declare my variable as the right kind. What I want in the end is an "int", and I'm apparently stating with a String, but I don't know why the function is making things into chars of any kind, much less what the difference in these kinds is. Can you help?

#5

chupacabracito:
Unfortunately I keep getting the error: "invalid conversion from 'unsigned char' to 'const char*' ".

That would be a problem in your code, then. If you posted the complete code and the compiler output I expect you'd get suggestions about how to fix it.

#6

PaulS:
The results of the request ARE all the "garbage". That only a part of the response is of interest to you is rather inconsequential.

Ok, good to know. Too bad there's not a more direct method, but ok.

PaulS:
No, you aren't. HTTPoutputString is not a string. It's a String, which is a completely different beast.

Ok. Can you explain the difference? I barely understand the difference between String and char. Or, more importantly, what I should do about the difference?

PaulS:
Once you have the useless String, use toCharArray() to convert it to a useable string, then use strtok() to extract the tokens. The last one will (probably) be the one that you want. Use itoa() to convert that token to an int.

Except that as I said, I tried toCharArray() already and it didn't work, because of the errors mentioned above. For this, the error is: "incompatible types in assignment of 'const char [1]' to 'char [4]' " given the code below:

 while (www.connected()) {
            while (www.available()) {
              char c = www.read();  //Reads the output of the HTTP Get Request
              Serial.print(c);
              HTTPoutputString += c;  //concatenates each character onto the end of the string as it comes
              
              }
            }  
          www.close();
          
          char Buf[4];
          HTTPoutputString.toCharArray(Buf, 3);          
          //char Foo = HTTPoutputString.charAt(3);
                    
          // DIDN'T WORK:   Buf = HTTPoutputString.endsWith(6);
          // DIDN'T WORK:   HTTPoutputString.toCharArray(Buf, 3);
          // DIDN'T WORK:   strncopy(HTTPoutputString, Buf, 3);
          // DIDN'T WORK:   HTTPoutputString.getBytes(Buf,1);
          // DOESNT'T WORK:  memcpy( HTTPoutputString, readString, strlen(readString)+1 );  ;
          //HTTPoutputString = readString; 
          //SetpointHeat = HTTPoutputString;
          Serial.print("\nHTTPoutputString: ");   Serial.print(HTTPoutputString);  Serial.print("\n");
          Serial.print("\n\nBuf: ");   Serial.print(Buf);  Serial.print("\n");
          //Serial.print("\n\nFoo: ");   Serial.print(Foo);  Serial.print("\n");
          Serial.println(F("-------------------------------------"));
          HTTPoutputString = "";
          Buf = "";

I also tried your strtok suggestion, and couldn't get it to work. I got the error "cannot convert 'uint8_t*' to 'char**' for argument '3' to 'char* strtok_r(char*, const char*, char**)' " with the following code:

 while (www.connected()) {
            while (www.available()) {
              char c = www.read();  //Reads the output of the HTTP Get Request
              Serial.print(c);
              HTTPoutputString += c;  //concatenates each character onto the end of the string as it comes
              
              }
            }  
          www.close();
          
          char Buf[3];
          HTTPoutputString.toCharArray(Buf, 200);
          char *BufFoo = strtok_r(Buf,"HTTP",&i);
          char Foo = HTTPoutputString.charAt(166);   //different tactic, also not working
                    
          // DIDN'T WORK but at least compiled:   char *BufFoo = strtok(Buf,"HTTP");
          // DIDN'T WORK:   Buf = HTTPoutputString.endsWith(6);
          // DIDN'T WORK:   HTTPoutputString.toCharArray(Buf, 3);
          // DIDN'T WORK:   strncopy(HTTPoutputString, Buf, 3);
          // DIDN'T WORK:   HTTPoutputString.getBytes(Buf,1);
          // DOESNT'T WORK:  memcpy( HTTPoutputString, readString, strlen(readString)+1 );  ;
          //HTTPoutputString = readString; 
          //SetpointHeat = HTTPoutputString;
          Serial.print("\nHTTPoutputString: ");   Serial.print(HTTPoutputString);  Serial.print("\n");
          Serial.print("\nBuf: ");   Serial.print(Buf);  Serial.print("\n");
          Serial.print("\nFoo: ");   Serial.print(Foo);  Serial.print("\n");
          Serial.print("\nBufFoo: ");   Serial.print(Foo);  Serial.print("\n");
          Serial.println(F("-------------------------------------"));
          HTTPoutputString = "";
          Buf[0] = 0; Buf[1] = 0; Buf[2] = 0;

Suggestions?

#7

If the returned data is simple and always in a known format, then simple parsing code like below might be used. There are more complex examples of client parsing code around for weather pages and such.

//zoomkat 12-16-11
//simple client test
//for use with IDE 1.0
//open serial monitor and send an e to test
//for use with W5100 based ethernet shields

#include <SPI.h>
#include <Ethernet.h>

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; //physical mac address

char serverName[] = "web.comporium.net"; // zoomkat's test web page server
EthernetClient client;


String readString, readString1;
int x=0;
char lf=10;
//////////////////////

void setup(){

  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    while(true);
  }

  Serial.begin(9600); 
  Serial.println("Better client test 9/22/12"); // so I can keep track of what is loaded
  Serial.println("Send an e in serial monitor to test"); // what to do to test
}

void loop(){
  // check for serial input
  if (Serial.available() > 0) //if something in serial buffer
  {
    byte inChar; // sets inChar as a byte
    inChar = Serial.read(); //gets byte from buffer
    if(inChar == 'e') // checks to see byte is an e
    {
      sendGET(); // call sendGET function below when byte is an e
    }
  }  
} 

//////////////////////////

void sendGET() //client function to send/receive GET request data.
{
  if (client.connect(serverName, 80)) {  //starts client connection, checks for connection
    Serial.println("connected");
    client.println("GET /~shb/arduino.txt HTTP/1.1"); //download text
    client.println("Host: web.comporium.net");
    client.println("Connection: close");  //close 1.1 persistent connection  
    client.println(); //end of get request
  } 
  else {
    Serial.println("connection failed"); //error message if no client connect
    Serial.println();
  }

  while(client.connected() && !client.available()) delay(1); //waits for data
  while (client.connected() || client.available()) { //connected or data available
    char c = client.read(); //gets byte from ethernet buffer
    Serial.print(c); //prints byte to serial monitor 
    if (c==lf) x=(x+1);
    if (x==9) readString += c;
  }

    Serial.println();  
    Serial.println();
    Serial.print("Current data row:" );
    Serial.print(readString);
    Serial.println();
    readString1 = (readString.substring(0,8));
    Serial.println();
    Serial.print("How we feeling?: ");
    Serial.println(readString1);
    Serial.println();      
    Serial.println("done");
  Serial.println("disconnecting.");
  Serial.println("==================");
  Serial.println();
  client.stop(); //stop client

}
#8

Suggestions?

Yes, several.

  1. Take some time to THINK.
  2. Look at the error messages. They tell you exactly what is wrong. If you have some code that produces an error message that you don't understand, post the code, and the error message. You said, for instance, that you tried to use the toCharArray() method and that that didn't work. And, yet, the error message is complaining about some other code.
  3. When a suggestion is made to use a function, use that function, not some other function.

I also tried your strtok suggestion, and couldn't get it to work.

          char *BufFoo = strtok_r(Buf,"HTTP",&i);

There is no reason to use the thread-safe, re-entrant version of strtok on a single-threaded system. Use the strtok() function, which is far simpler, like you were told.

#9

zoomkat:
If the returned data is simple and always in a known format, then simple parsing code like below might be used. There are more complex examples of client parsing code around for weather pages and such.

Sweet, thanks, Zoomkat! That example totally helped. I was able to use your method to get just the data I wanted in the String, and then some searching on other forums showed me the .toInt() command to convert the String to an int. (For anyone in the future reading this, it's easy: if you have a String called "Foo", you can convert it to an int by writing "Foo.toInt()".

It works now! Very much appreciated.

#10

PaulS:

Suggestions?

Yes, several.

  1. Take some time to THINK.
  2. Look at the error messages.

PaulS, as much as I appreciate your responses, let me give you some feedback on how you write: when you say things like "take some time to THINK" and "look at the error messages" and "No, you aren't. HTTPoutputString is not a string. It's a String, which is a completely different beast", I read those statements as sounding very condescending, and I felt more insulted than helped by them. For your #3 comment, you're right that the code snippet I pasted used a variant of the function you mentioned--I failed to clarify that after I tried exactly the function you mentioned, and it didn't work, I tried four other things as well that I found on different forums, and the last one happened to be that variant in the code snippet I pasted. So that was me being unclear, sorry. But in general, in the future, you might want to consider how your writing comes off to the people you're trying to help. I, personally, did not react positively to your posts.

#11

But in general, in the future, you might want to consider how your writing comes off to the people you're trying to help. I, personally, did not react positively to your posts.

You might consider how much you paid for the help you got, and how much you said, or failed to say, about what you tried and what happened.

The compiler doesn't generate random error messages. Taking some time to think about what the compiler is telling you pays off in the long run. Taking some time to actually read the error messages, and trying to understand how that relates to the input you gave it, pays off in the long run.

Instead of "that didn't work", you need to say exactly what you did, and exactly what actually happened. YOU could have helped yourself tremendously by learning how to write the code that zoomkat spoon fed you. He may not be around to hold your hand next time.

And, by the way, I really don't mind if you don't like this answer, either. I'll still sleep well tonight.

#12

Whenever I think PaulS is a pain in the arse (or ass, depending on where you live), which is probably a few times a day week, I remember this…

paulS.PNG

#13