Ignore http GET data after the useful data.

I'm trying to read some thingspeak data from a field. I get the data succesfully, but alot of the http junk before the useful data.

What's the easiest way to strip the stuff before the useful data?

The value returned will vary in size from 1 character, to 5 characters.
Well, actually this part of the sketch will be replicated for other channels/fields. Each channel/field will be the same number of characters.

So MBRstatus will always be a 0 or 1.
(not listed yet) MBRtemp will always be 2 characters

etc etc etc.

This is the serial output ("1" is the only useful data, at the end)

...WiFi connected
IP Address: 192.168.0.200
HTTP/1.1 200 OK

Content-Type: text/html; charset=utf-8

Content-Length: 4

Connection: close

Status: 200 OK

X-Frame-Options: ALLOWALL

Access-Control-Allow-Origin: *

Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH

Access-Control-Allow-Headers: origin, content-type, X-Requested-With

Access-Control-Max-Age: 1800

ETag: "*********************************"

Cache-Control: max-age=0, private, must-revalidate

X-Request-Id: ****************************

X-Runtime: 0.007634

X-Powered-By: Phusion Passenger 4.0.57

Date: Wed, 21 Sep 2016 03:09:56 GMT

Server: nginx/1.9.3 + Phusion Passenger 4.0.57



1

Here's the sketch.

// D0=16, D1=5, D2=4, D3=0, D4=2, D5=14, D6=12, D7=13, D8=15
#include <ESP8266WiFi.h>

const char* ssid          = "WiFi";  
const char* password      = "12345678";



// Master Bedroom Stuff
const int MBRservoPin         = 5;   //Master Bedroom - D1
const int MBRopenPos          = 0;
const int MBRclosePos         = 40;
const char* MBRhost           = "api.thingspeak.com";
const char* MBRchannel       = "000000";
const char* MBRfield          = "1";
const char* MBRreadAPI        = "KEY GOES HERE";
String MBRstatus              = "";

void setup() {
  Serial.begin(115200);
  delay(10);
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);
  int wifi_ctr = 0;
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("WiFi connected");  
  Serial.print("IP Address: ");
  Serial.println(WiFi.localIP());
}

void loop() {
  WiFiClient client;
  const int httpPort = 80;
  if (!client.connect(MBRhost, httpPort)) {
    Serial.println("connection failed");
    return;
  }
  String MBRurl = "/channels/";
         MBRurl += MBRchannel;
         MBRurl += "/fields/";
         MBRurl += MBRfield;
         MBRurl += "/last?api_key=";
         MBRurl += MBRreadAPI;
         
  client.print(String("GET ") + MBRurl + " HTTP/1.1\r\n" +
               "Host: " + MBRhost + "\r\n" + 
               "Connection: close\r\n\r\n");
               
  delay(250);   //wait for server to respond
  while(client.available()){
    MBRstatus = client.readStringUntil('\r');
    Serial.println(MBRstatus);
  }


  delay(5000);

}

Thank you!

What's the easiest way to strip the stuff before the useful data?

The easiest way is to not store the useless data. It appears that the "useful" data is preceded by two carriage return/line feed combination. So, if a character is not a CR, then you have not yet reached the useful data. If it is, then maybe you are getting close. If the next character is not a LF, you are not.

If the next character is not a CR, you have reached the start of a new record of "useless" data. Repeat reading and dumping data until you see CR, LF, CR, LF. When you see the second LF, with nothing before it but the CR, LF, CR sequence, start saving the data.

The HTTP protocol is defined here: https://www.ietf.org/rfc/rfc2616.txt.

Short answer - the header lines are terminated with a CRLF, and they finish with a blank line, after which is the data.

However, you should always attempt to parse the Content-Length header, as an HTTP message can terminate with closing the channel, or a server can send HTTP messages back-to-back by giving the length of the data. If you don't do this, then you must send a 'Connection: close' header with your request so that the server knows to signal end of data by closing the tcp connection.

Incase anybody finds this thread and has the same issue.... just use the ThingSpeak library. Didn't know one existed and wow it makes it sooo much easier...