Adafruit CC3000 WiFi Shield + HTTP GET request + char + String = ARRRGGHHH!

Please, PLEASE help.

I'm losing my mind here.

I've got some programming experience (although I am currently rusty) but I am an Arduino and IoT beginner and know next to nothing about HTTP requests, so please be lenient with your responses. I've tried looking all over for a solution but have found nothing I can use.

I recently purchased the Adafruit CC3000 WiFi shield for my Arduino Uno in order to start building a project for my wife: a weather "station" where basically details about the weather pulled from a weather website will be displayed on the screen when she presses a button.

I have managed to successfully connect the shield and get it online and everything works fine. I even managed to get the information I needed from the GET request about the weather.

I modified the "WebClient" example from the shield's library to achieve this.

The trouble comes when I try to get the incoming chars that are read from the GET request (they are printed one by one) and get them into a String in order to be able to later parse that String and get the information I need.

I don't know if it's a timing issue with the request or if there is a character that somehow makes the concatenation of the char (converted to String) to my main String break, but the String that I end up with in my variable is not complete.

This is the part of the code in question (where the concatenation happens as the characters are read in). The String I want to concatenate everything to is dataprint and the character read from the server while the connection is open is c.

Adafruit_CC3000_Client www = cc3000.connectTCP(ip, 80);
  if (www.connected()) {
    www.fastrprint(F("GET "));
    www.fastrprint(url);
    www.fastrprint(F(" HTTP/1.1\r\n"));
    www.fastrprint(F("Host: ")); www.fastrprint(WEBSITE); www.fastrprint(F("\r\n"));
    www.fastrprint(F("\r\n"));
    www.println();
  } else {
    Serial.println(F("Connection failed"));    
    return;
  }

  Serial.println(F("-------------------------------------"));

  /* Read data until either the connection is closed, or the idle timeout is reached. */ 
  unsigned long lastRead = millis();
  while (www.connected() && (millis() - lastRead < IDLE_TIMEOUT_MS)) {
    while (www.available()) {
      char c = www.read();

        
      dataprint.concat(String(c));
      Serial.print(c);
      lastRead = millis();
    }
  }
   
  www.close();

In order to demonstrate the "breaking" that I am talking about, here is the HTTP GET request as printed on Serial when each individual char comes in:

HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Fri, 27 Feb 2015 22:13:13 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Status: 200 OK
X-Frame-Options: ALLOWALL
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: "06d218cfbd8a255b22db2786373c43d5"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: bff09e30-f16c-4010-b219-b3df63f39a3b

312

                          <div class="info city-fcast-text">
                            <h3>Today</h3>
                            <h4>Feb 28</h4>
                            <div class="icon i-14-m "></div>
                            <span class="cond">Partly sunny, showers around</span> <span class="hi-low">Hi</span> <strong class="temp">16<span>°</span></strong>

                            <span class="realfeel">RealFeel 15°</span> </div>
                          <!-- /.info -->
                          <a href="http://www.accuweather.com/en/gr/athens/182536/daily-weather-forecast/182536?day=1" class="bt-more">more</a>
                          <div class="overlay o-s"></div>
                          <div class="frame"></div>
                        
0

...and here is how it looks when I do Serial.println(dataprint); :

HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Fri, 27 Feb 2015 22:13:13 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Status: 200 OK
X-Frame-Options: ALLOWALL
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: "06d218cfbd8a255b22db2786373c43d5"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: bff09e30-f16c-4010-b219-b3df63f39a3b

312

                          <div class="info city-fcast-text">
                            <h3>Today</h3>
                            <h4>Feb 28</h4>
                            <div class="icon i-14-m "></div>

//gibberish characters appear here that can't be copy-pasted
     ŒŒŒ  ŒŒŒŒ

Does anyone know of a better way to do this? Can anyone please help me with what's wrong? I'm sure I have a lot of work ahead of me with parsing the information (since the actual request I want to make contains more information and is more complicated) but I'm already stuck here, so I could use someone's guidance to finally move on.

Thank you so much in advance. I attach the entire sketch for anyone who has the same shield I do and can test this. Please excuse the silly function names. A man's gotta do something to make things lighter when he's close to losing his mind.

WebClient_H.ino (5.45 KB)

I'm losing my mind here.

Yep. Right here:

      dataprint.concat(String(c));

First, the concat() method is overloaded to take a character. There is no need to waste memory constructing, unwrapping, and destructing a String instance to add a character to a String.

Second, you are making Swiss cheese of your memory by using the String class at all. Stop that.

You are clearly running out of memory.

for this_line in lines:
if this_line.startswith("<a href"):
bits = this_line.split('"')
print bits[1]

Python

what's got
todo with
the question ?

Monty

PaulS:
Yep. Right here:

      dataprint.concat(String(c));

First, the concat() method is overloaded to take a character. There is no need to waste memory constructing, unwrapping, and destructing a String instance to add a character to a String.

Second, you are making Swiss cheese of your memory by using the String class at all. Stop that.

You are clearly running out of memory.

Thank you for your reply.

I'd be happy to "stop that", and use something other than String, but would you please provide me with a better way to do this?

First of all, the concat() method would not accept a char when I tried it before (seems to be verifying but I need to run it again), and secondly, I don't know how to append a char to a char array. I tried setting a large enough size for the array to contain the characters (5000 for the example I actually want to use) and doing something like this:

//(previously defined i)
chararray[i] = c;
i++;

...but it gave me errors when trying to initialize it and the program just would not start.

dataprint += c

Should be all that's needed.

If String is killing your memory, use char array. You're on the right track but misguided by not showing your code and not telling us what the error is.

Hello,

I have attached the sketch in the first post. You can download and see it. I didn't think dumping it would be considerate.

I have also described the error in the first post. The printed String does not correspond to what comes through one character at a time from the GET request.

Could you please tell me what the Arduino code is for appending to a char array, or alternatively making it so one can set its values via index? I have searched for some examples but C++ libraries and certain classes don't seem to apply to the Arduino environment.

There seems to be an issue with the size of the char array. I tried changing the String dataprint; to char dataprint[6000];, defining int i = 0; and then using this code:

Adafruit_CC3000_Client www = cc3000.connectTCP(ip, 80);
  if (www.connected()) {
    www.fastrprint(F("GET "));
    www.fastrprint(url);
    www.fastrprint(F(" HTTP/1.1\r\n"));
    www.fastrprint(F("Host: ")); www.fastrprint(WEBSITE); www.fastrprint(F("\r\n"));
    www.fastrprint(F("\r\n"));
    www.println();
  } else {
    Serial.println(F("Connection failed"));    
    return;
  }

  Serial.println(F("-------------------------------------"));

  /* Read data until either the connection is closed, or the idle timeout is reached. */ 
  unsigned long lastRead = millis();
  while (www.connected() && (millis() - lastRead < IDLE_TIMEOUT_MS)) {
    while (www.available()) {
      char c = www.read();
      dataprint[i] = c;                        //CHANGED
      i++;                                        //CHANGED
      Serial.print(c);
      lastRead = millis();
    }
  }
   
  www.close();

...but if I define char dataprint[6000]; it hangs completely and never starts. If I define it as something smaller, it reaches the point where the index ends and then hangs there, not even printing the GET request.

Is this simply an impossibility for the amount of characters I want to get from the GET request? Is it a limitation of the language/hardware? I want to learn, so please help me understand; I don't think an admonishing tone ever helps further the community or really teach anyone how to advance.

but if I define char dataprint[6000]; it hangs completely and never starts.

No wonder, because you are trying to set aside 6000 bytes of memory for the array. How much memory does a Uno have ?

For 6000 bytes you need a chip like '1284P with 16K SRAM
I offer boards in several form factors, such as this one
http://www.crossroadsfencing.com/BobuinoRev17/

Also how many characters do you think is this?

HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Fri, 27 Feb 2015 22:13:13 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Status: 200 OK
X-Frame-Options: ALLOWALL
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: "06d218cfbd8a255b22db2786373c43d5"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: bff09e30-f16c-4010-b219-b3df63f39a3b

312

                          <div class="info city-fcast-text">
                            <h3>Today</h3>
                            <h4>Feb 28</h4>
                            <div class="icon i-14-m "></div>
                            <span class="cond">Partly sunny, showers around</span> <span class="hi-low">Hi</span> <strong class="temp">16<span>°</span></strong>

                            <span class="realfeel">RealFeel 15°</span> </div>
                          <!-- /.info -->
                          <a href="http://www.accuweather.com/en/gr/athens/182536/daily-weather-forecast/182536?day=1" class="bt-more">more</a>
                          <div class="overlay o-s"></div>
                          <div class="frame"></div>
                        
0

If that's still too much only start storing char in the buffer when you see '<': mark of the beginning of the html body.

Or you can read and parse one line of html at a time looking for the '\n' (newline) as the delimiter.

UKHeliBob:
No wonder, because you are trying to set aside 6000 bytes of memory for the array. How much memory does a Uno have ?

I'm not sure, which is why I'm asking. If 6000 bytes are not a reality and I can't use String, am I basically unable to make this project happen, do you think? Is there a better way to get weather data from the internet rather than do a GET request through ThingSpeak? I am asking because I haven't found an example or any guidance for usage with the Adafruit CC3000 WiFi shield. Most sketches and libraries that would make this easier are written for the Ethernet shields.

The WiFi shield has a microSD card reader on it. Would it be possible to use that as more memory for the sketch or is it only for other data storage or logging? I haven't even gotten to that part of the shield so I'm not sure.

mistergreen:
Also how many characters do you think is this?

It's another example I want to work up to.
More specifically, it's this GET request:

https://api.thingspeak.com/apps/thinghttp/send_request?api_key=HBW9W33TJFCBDQ7A

When I copied the result of the GET request off of Serial and put it in a character counting website, it gave me ~5000 chars.

EDIT: If I even manage to isolate/store only the part I'm interested in, it's still 2695 characters. :confused:

mistergreen:
If that's still too much only start storing char in the buffer when you see '<': mark of the beginning of the html body.

Thank you! That might help reduce it a little bit. It still probably won't save me from having to create a big char array, though.

mistergreen:
Or you can read and parse one line of html at a time looking for the '\n' (newline) as the delimiter.

Would I do that using String? If not, how would I go about doing it with a char array?

Something like

dataPrint[200];
......

    while (www.available()) {

      char c = www.read();
      if(c == '\n') {
        parseTheHtmlTagForValues(dataPrint);
        //clear out dataprint and start over
        memset(&dataprint[0], 0, sizeof(dataprint));
        i = 0;
      }
      dataprint[i] = c;                
      i++;                                       
      Serial.print(c);
      lastRead = millis();
    }

.....
void parseTheHtmlTagForValues(dataPrint) {
      char *cPointer;
      //find conditions tag, grab value in between
      if(cPointer = strstr(dataPrint, "<span class=\"cond\">")) {
           cPointer += 19;
      }

        char *s = cPointer;
        char *value_start_pos = s;
        int val_len = 0;

        //< marks the end of the closing html tag. stop looking there.
        while(*s != '<') {
            val_len++;
            s++;
            
        }

        char in_holder[100]; // temporary holder.... copy what you find into it.
        strncpy(in_holder, value_start_pos, val_len);
        in_holder[val_len+1] = '\0'; // end char string properly

       Serial.println(in_holder);
      // do whatever with in_holder, make it global etc...


}

Which part of the captured html do you need? Below is some old test code that counts carriage returns to find the incoming line to capture from all the raw data sent. The below textfinder application might be of use.

http://playground.arduino.cc/Code/TextFinder

//zoomkat 12-22-10
//simple ethernet client test code
//for use with IDE 0021 and W5100 ethernet shield
//modify the arduino lan ip address as needed
//open serial monitor to see what the arduino receives
//push the shield reset button to run client again

#include <SPI.h>
#include <Ethernet.h>
String readString, readString1;
int x=0;
char lf=10;

byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
//byte ip[] = { 192, 168, 1, 102 };
//byte server[] = { 140, 90, 238, 27 }; // NOAA
char serverName[] = "www.ndbc.noaa.gov";

EthernetClient client;

void setup()
{
  Ethernet.begin(mac);
  Serial.begin(9600);
  Serial.println("starting simple arduino client test");
  Serial.println();
  Serial.println("connecting...");

  if (client.connect(serverName, 80)) {
    Serial.println("connected");
    client.println("GET /data/5day2/44013_5day.txt HTTP/1.1");
    client.println("Host: www.ndbc.noaa.gov");
    client.println("Connection: close"); 
    client.println();
  } else {
    Serial.println("connection failed");
  }
}

void loop()
{
  if (client.available()) {
    char c = client.read();
    //Serial.print(c);  // uncomment to see raw feed
    if (c==lf) x=(x+1);
    if (x==14) readString += c;
    //readString += c;
  }

  if (!client.connected()) {
     client.stop();

    Serial.println("Current data row:" );
    Serial.print(readString);
    Serial.println();
    readString1 = (readString.substring(41,43));
    Serial.println();
    Serial.print("DPD sec: ");
    Serial.println(readString1);
    Serial.println("done");

    for(;;);

    }
 }

Thank you both so much for your replies! I will play around with the (test) code and come back with results...

zoomkat, from this GET request: https://api.thingspeak.com/apps/thinghttp/send_request?api_key=HBW9W33TJFCBDQ7A

I need from the beginning of where it says "Currently" to the end of "Tomorrow" (although that changes so the latter might be tonight; basically three out of four temp + condition readings).

I guess if I use yours and/or mistergreen's code I could carefully parse so I minimize the amount of characters needed as much as possible, but it still is looking a little grim. I searched a bit online and it doesn't seem to be possible to use the SD card's memory for runtime execution, but some were able to write Strings to it.

To be continued...

zoomkat, from this GET request: Sign In - ThingSpeak IoT

The "s" in "https" is may stop your project as I don't think the secure sockets thing is supported in the arduino ethernet libraries.

I don't think the secure sockets thing is supported in the arduino ethernet libraries.

It is not. Supporting it takes too much memory.

The https part hasn't presented a problem for me. I just realized there is no way to realistically fit the big request in one String and process it, so I am making four smaller requests instead, re-using the dataprint String. I am using your ideas and turning the String concatenation on once the HTML starts but not reading anything that is within the html tags. I'll find a way to clean up the extra spaces and dashes and extract the information after this.

Thank you so much for your guidance. It helped me get unstuck.

  /* Read data until either the connection is closed, or the idle timeout is reached. */ 
int x = 0;

  unsigned long lastRead = millis();
  while (www.connected() && (millis() - lastRead < IDLE_TIMEOUT_MS)) {
    while (www.available()) {
      char c = www.read();
      if(c == '>') {x = 1;}
      if(c == '<') {x = 0;}
      
      if(x == 1 && c != '>') 
      {   
          if(c == '\n')
          {dataprint += ' ';}
          else if (c == '\r')
          {}
          else
          {dataprint += c;}
      }
      Serial.print(c);
      lastRead = millis();
    }

   
  www.close();
  x = 0;
  }

A request that prints on Serial like this:

HTTP/1.1 200 OK
Server: nginx/1.7.5
Date: Sun, 01 Mar 2015 20:53:23 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
Status: 200 OK
X-Frame-Options: ALLOWALL
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET, POST, PUT, OPTIONS, DELETE, PATCH
Access-Control-Allow-Headers: origin, content-type, X-Requested-With
Access-Control-Max-Age: 1800
ETag: "1738e5e96d3a00e0f4558ffe4dd7d64b"
Cache-Control: max-age=0, private, must-revalidate
X-Request-Id: 770d9304-9664-4b62-818e-db264905fbb6

177

                            <h3>Tonight</h3>
                            <h4>Mar 1</h4>
                            <div class="icon i-34-m "></div>
                            <span class="cond">Mainly clear</span> <span class="hi-low">Lo</span> <strong class="temp">8<span>°</span></strong>

                            <span class="realfeel">RealFeel 8°</span> 
0

...becomes this in dataprint:

Tonight Mar 1 Mainly clear Lo 8° RealFeel 8° 0

...albeit with more spaces inbetween that I need to erase.

Whipped up this cleaning function for the spaces that adds dashes if there is more than one space (indicating a different piece of data). Next is being able to parse this (either in the same function or in another one) in order to display the different pieces of data on an LCD screen.

String clearString(String datString){
  int i = 0;
  int y = 0;
    String tempString;
    while (i < datString.length()) {
        if(datString[i] == ' '){
          y++;
          if (y > 1){
            if (y == 2){
              tempString += "- ";
            }
            else{}
          }
          else {
            tempString += datString[i];
          }
        }
        
        else {
        y = 0;
        tempString += datString[i];
        }
        
        i++;
    }
  
  return tempString;  
}

...it ends up looking like this (with the useless 0 and - in the end). Need to find a way to separate the condition and the temp since I'm not reading within the tags and it doesn't have a set length. Maybe something like indexOf() and then adding a dash before that...

Tuesday - Mar 3 - Mostly sunny and nice Hi 18° - RealFeel 19° - 0 -