Capturing string from GET request Ethernet Shield

I’m fairly new to programming so i apologize if the answer is obvious but I have been stuck on this for several days now. After much research around the forum I am still having problems capturing a GET request in string form from the client.read() function from an ethernet shield. I understand client.read() reads the data byte by byte but every attempt I make to create a string out of the output characters simply doesnt work even with multiple different ways that have been posted on here as solutions (char array, String format). I posted my code below and cannot for the life of me figure out why it is not working for me! if anyone could point me in the right direction it would be greatly appreciated. Basically the end goal is to have the entire GET request in a single char array or String format that can be used fro string comparison and to extract data. When I run the code below the output for Serial.print(readString); is either blank or [1200] and it seems that the readString or tBuffer does not have any of the data from the GET request.

 Circuit:
 * Ethernet shield attached to pins 10, 11, 12, 13
 * RGB pins 3,5,6
 Underlying Ethernet Code:
 created 18 Dec 2009
 by David A. Mellis
 modified 9 Apr 2012
 by Tom Igoe, based on work by Adrian McEwen
 
 */

#include <SPI.h>
#include <Ethernet.h>
#include <WString.h>


// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = { 0x00, 0x0A, 0x95, 0x9A, 0x68, 0x16 };
// if you don't want to use DNS (and reduce your sketch size)
// use the numeric IP instead of the name for the server:
//IPAddress server(98,137,236,150);  // numeric IP for Google (no DNS)
char server[] = "www.finance.google.com";    // name address for Google (using DNS)

// Set the static IP address to use if the DHCP fails to assign
IPAddress ip(10,0,0,22);

// Initialize the Ethernet client library
// with the IP address and port of the server 
// that you want to connect to (port 80 is default for HTTP):
EthernetClient client;



void setup() {
 // Open serial communications and wait for port to open:
  Serial.begin(9600);


  // start the Ethernet connection:
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    // no point in carrying on, so do nothing forevermore:
    // try to congifure using IP address instead of DHCP:
    Ethernet.begin(mac, ip);
  }
  // give the Ethernet shield a second to initialize:
  
  Serial.println("connecting...");

  // if you get a connection, report back via serial:
  if (client.connect(server, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    client.println("GET /finance/info?q=NASDAQ:GOOG HTTP/1.1\r\n");
    client.println("Host: www.finance.google.com");
    client.println("Connection: close");
    client.println();
  } 
  else {
    // kf you didn't get a connection to the server:
    Serial.println("connection failed");
  }
}

void loop()
{


String readString = String(1200);
readString = "";

  char tBuffer[1200];
  int tCount = 0;
  
  
  if (client.available()) {
 
    char c = client.read();
   Serial.print(c);
   
  /*if (tCount < 1199){        //    First attempt to create char array to store incoming client data unsuccessful
    tBuffer[tCount] = c;
    tCount++;
    tBuffer[tCount] = 0;
  }
  */
  //Client.read data appended to readString to create string that can be used for strcmp
 readString += c;
 Serial.println(readString);
 //readString displays same thing as c 
  }
  

  // if the server's disconnected, stop the client:
  if (!client.connected()) {
    Serial.println();
    Serial.println("disconnecting.");
    client.stop();
Serial.print(readString);
 
    // do nothing forevermore:
    while(true);
  }
 
}

Chances are you are running out of SRAM. Which Arduino are you using?

jtwalk3r: When I run the code below the output for Serial.print(readString); is either blank or [1200] and it seems that the readString or tBuffer does not have any of the data from the GET request.

You are very optimistic about RAM in your sketch when using

String readString = String(1200);

Which Arduino board are you using? How long do you expect the GET request to be? Do you really expect a GET request 1200 characters long?

I am using the arduino Uno, admittedly I completely forgot to think about SRAM size and I think I just arbitrarily threw in the 1200 without thinking about it too much.

When I run the code below the output for Serial.print(readString); is either blank or [1200] and it seems that the readString or tBuffer does not have any of the data from the GET request.

Well, the 1200 is easy to explain.

String readString = String(1200);

This is identical to

String readString = "1200";

I fail to see how that is a useful initial value.

I second Tim's suggestion that you are running out of memory. Learn to use the F() macro: Serial.println(F("Failed to configure Ethernet using DHCP"));

Finally, your post rambled on about strings, and then you dump this code here with those stinking Strings in it. The String class is NOT the same as a C string.

jurs: How long do you expect the GET request to be? Do you really expect a GET request 1200 characters long?

It isn't a GET request the OP is storing. It is the response to a GET request, which from any Google site page request, will be much longer than 1200 characters.

You will probably need to wait for a particular part of the response before storing the characters.

That array may use more SRAM than your Arduino has available.

You have to determine what string in the response you are looking for, and wait for the part that you want before storing anything.

edit: Whoopsie! The OP deleted the last post.

Still I was confused why the commented out attempt at creating a char array failed to work as well or should I jsut assume it is the memory problem?

It is most assuredly a memory problem. Reserving, arbitrarily, 1200 bytes of the 2048 that are available, does not make sense.

Ok that makes sense to me that im clearly using way too much memory. @SurferTim How would one go about doing that say if I wanted to record the data after "lastchg"? Would I just create a smaller buffer and then once the buffer = "lastchg" just record the client.read() data to a different character array?

@SurferTim Ok that makes sense to me that im clearly using way too much memory. How would one go about doing that say if I wanted to record the data after "lastchg"? Would I just create a smaller buffer and then once the buffer = lastchg just record the data to a different character array?

Well, now we have no idea what you are talking about...

Ok, I just am wondering how I would go about SurferTims suggestion to wait for a part of the response before I start storing data sorry for the confusion

You can normally disregard the header, so wait for a blank line. That will denote the end of the response header.

Then you need to determine a unique part of the response and wait for it. Normally I would read a line until a cr/lf, then check that line for the part you want, but Google web pages are notorious for very long lines.

The below is textfinder that extracts data from large amounts of data being received by the arduino. Bottom is code that extracts weather data from a weather web page that you might study.

http://playground.arduino.cc/Code/TextFinder

//////////////////////////////////////////////
// Get XML formatted data from the web.
// 1/6/08 Bob S. - Created
//  Assumptions: single XML line looks like: 
//    <tag>data</tag> or <tag>data 
// Include description files for other libraries used (if any)
//
// updated at later dates zoomkat
//////////////////////////////////////////////

#include <SPI.h>
#include <Ethernet.h>

// Define Constants
// Max string length may have to be adjusted depending on data to be extracted
#define MAX_STRING_LEN  20

// Setup vars
char tagStr[MAX_STRING_LEN] = "";
char dataStr[MAX_STRING_LEN] = "";
char tmpStr[MAX_STRING_LEN] = "";
char endTag[3] = {'<', '/', '\0'};
int len;

// Flags to differentiate XML tags from document elements (ie. data)
boolean tagFlag = false;
boolean dataFlag = false;

// Ethernet vars
byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
//byte ip[] = { 192, 168, 1, 102 };
byte server[] = { 140, 90, 113, 200 }; // www.weather.gov

// Start ethernet client
EthernetClient client;

void setup()
{
  Serial.begin(9600);
  Serial.println("Starting WebWx");
  Serial.println("connecting...");
  //Ethernet.begin(mac, ip);
  Ethernet.begin(mac);
  delay(1000);

  if (client.connect(server, 80)) {
    Serial.println("connected");
    client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
    client.println();
    delay(2000);
  } else {
    Serial.println("connection failed");
  }  
}

void loop() {

  // Read serial data in from web:
  while (client.available()) {
    serialEvent();
  }

  if (!client.connected()) {
    Serial.println();
    Serial.println("Disconnected");
    Serial.println("==================================");
    Serial.println("");
    client.stop();

    // Time until next update
    //Serial.println("Waiting");
    for (int t = 1; t <= 15; t++) {
      delay(60000); // 1 minute
    }

    if (client.connect(server, 80)) {
      //Serial.println("Reconnected");
      client.println("GET /xml/current_obs/KRDU.xml HTTP/1.0");    
      client.println();
      delay(2000);
    } else {
      Serial.println("Reconnect failed");
    }      
  }
}

// Process each char from web
void serialEvent() {

   // Read a char
	 char inChar = client.read();
   //Serial.print(".");
  
   if (inChar == '<') {
      addChar(inChar, tmpStr);
      tagFlag = true;
      dataFlag = false;

   } else if (inChar == '>') {
      addChar(inChar, tmpStr);

      if (tagFlag) {      
         strncpy(tagStr, tmpStr, strlen(tmpStr)+1);
      }

      // Clear tmp
      clearStr(tmpStr);

      tagFlag = false;
      dataFlag = true;      
      
   } else if (inChar != 10) {
      if (tagFlag) {
         // Add tag char to string
         addChar(inChar, tmpStr);

         // Check for </XML> end tag, ignore it
         if ( tagFlag && strcmp(tmpStr, endTag) == 0 ) {
            clearStr(tmpStr);
            tagFlag = false;
            dataFlag = false;
         }
      }
      
      if (dataFlag) {
         // Add data char to string
         addChar(inChar, dataStr);
      }
   }  
  
   // If a LF, process the line
   if (inChar == 10 ) {

/*
      Serial.print("tagStr: ");
      Serial.println(tagStr);
      Serial.print("dataStr: ");
      Serial.println(dataStr);
*/

      // Find specific tags and print data
      if (matchTag("<temp_f>")) {
	      Serial.print("Temp: ");
         Serial.print(dataStr);
      }
      if (matchTag("<relative_humidity>")) {
	      Serial.print(", Humidity: ");
         Serial.print(dataStr);
      }
      if (matchTag("<pressure_in>")) {
	      Serial.print(", Pressure: ");
         Serial.print(dataStr);
         Serial.println("");
      }

      // Clear all strings
      clearStr(tmpStr);
      clearStr(tagStr);
      clearStr(dataStr);

      // Clear Flags
      tagFlag = false;
      dataFlag = false;
   }
}

/////////////////////
// Other Functions //
/////////////////////

// Function to clear a string
void clearStr (char* str) {
   int len = strlen(str);
   for (int c = 0; c < len; c++) {
      str[c] = 0;
   }
}

//Function to add a char to a string and check its length
void addChar (char ch, char* str) {
   char *tagMsg  = "<TRUNCATED_TAG>";
   char *dataMsg = "-TRUNCATED_DATA-";

   // Check the max size of the string to make sure it doesn't grow too
   // big.  If string is beyond MAX_STRING_LEN assume it is unimportant
   // and replace it with a warning message.
   if (strlen(str) > MAX_STRING_LEN - 2) {
      if (tagFlag) {
         clearStr(tagStr);
         strcpy(tagStr,tagMsg);
      }
      if (dataFlag) {
         clearStr(dataStr);
         strcpy(dataStr,dataMsg);
      }

      // Clear the temp buffer and flags to stop current processing
      clearStr(tmpStr);
      tagFlag = false;
      dataFlag = false;

   } else {
      // Add char to string
      str[strlen(str)] = ch;
   }
}

// Function to check the current tag for a specific string
boolean matchTag (char* searchTag) {
   if ( strcmp(tagStr, searchTag) == 0 ) {
      return true;
   } else {
      return false;
   }
}

There are probably multiple errors in this but from what i gather this is the general concept of what I should do.

 int i;
  char tBuffer[8];
  int tCount = 0;
  char final[] = "response";
  
  if (client.available()) {
 
    char c = client.read();
   Serial.print(c);
   
//Load buffer till full

  if (tCount < 8){        
    tBuffer[tCount] = c;
    tCount++;
    tBuffer[tCount] = 0;
  }

//Once full check if response has been found if not move one character over 

  if (tCount = 8){
    if(tBuffer == "response");
    {
      do something;
    }
    else{
     for ( i = 0; i < 8; i++){
      tBuffer[i] = tBuffer[i+1]; 
     }
     tBuffer[8] = c;
    }
  }

Edit: Posted before I saw Zoomkats code which indeed does look more promising

Try a variation of zoomkat's code. That looks more promising.

jtwalk3r:
There are probably multiple errors in this but from what i gather this is the general concept of what I should do.

The GET request is typically in the first line of the HTTP header.

If you just want that, here is some example code that:

  • reads the first line of an HTTP request
  • echo the request to Serial
  • echo the request back to the caller as “plain/text”
#include <SPI.h>
#include <Ethernet.h>
char linebuf[81]; // buffer for line length 80 and terminating '\0' character

// the next two lines must contain unique MAC and IP address in your LAN network
byte mac[] = { 0x00, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 2, 250 };

EthernetServer server(80);

char* readGetRequest(EthernetClient &client)
{
  int charcount=0;
  memset(linebuf,0,sizeof(linebuf));
  // http GET request is first line of request and 
  // ends with a '\n' newline character
  while (client.connected()) 
  {
    if (client.available()) 
    {
      char c = client.read();
      if (charcount<sizeof(linebuf)-1)
      {
        linebuf[charcount]=c;
        charcount++;
      }
      if (c == '\n') break; // end of header found, break while-loop
    }
  }
  return linebuf;
} // void readClientRequest(EthernetClient &client)

void sendEchoResponse(EthernetClient &client, char* text)
{
  client.println(F("HTTP/1.1 200 OK"));
  client.println(F("Content-Type: text/plain"));
  client.println();
  client.println(text);
}

void setup(){
  Serial.begin(9600);
  Ethernet.begin(mac, ip);
  Serial.print("Server is at: ");Serial.println(Ethernet.localIP());
}

void loop()
{
  EthernetClient client = server.available();
  if (client) 
  {
    char* request=readGetRequest(client);
    Serial.print("Got HTTP Request: ");  
    Serial.println(request);
    sendEchoResponse(client, request);
    client.flush();
    client.stop();
  }
}

@jurs: That is great if the OP is using a server sketch. The OP is using a client sketch to access a Google server, and trying to parse a string from the response.

char server[] = "www.finance.google.com";    // name address for Google (using DNS)

SurferTim:
@jurs: That is great if the OP is using a server sketch. The OP is using a client sketch to access a Google server, and trying to parse a string from the response.

I see. My mistake.

In that case he could use this code to:

  • send the request to the server
  • skip the response header
  • read in the response line by line

Example code showing numbered response lines in the serial monitor:

#include <SPI.h>
#include <Ethernet.h>
char linebuf[81]; // buffer for line length 80 and terminating '\0' character

// the next two lines must contain unique MAC and IP address in your LAN network
byte mac[] = { 0x00, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 192, 168, 2, 250 };

char server[]="finance.google.com";

void sendGetRequest(EthernetClient &client)
{
  // Make a HTTP request:
  client.println("GET /finance/info?q=NASDAQ:GOOG HTTP/1.1");
  client.print("Host: ");client.println(server);
  client.println("Connection: close");
  client.println("Connection: close");
  client.println();
}

void skipHeader(EthernetClient &client)
{ // skip response header from server
  boolean currentLineIsEmpty=true;
  while (client.connected()) 
  {
    if (client.available()) 
    {
      char c = client.read();
      if (c=='\n' && currentLineIsEmpty) break;
      else if (c=='\n') currentLineIsEmpty=true;
      else if (c>=' ') currentLineIsEmpty=false;
    }
  }
}
 
char* readClientLine(EthernetClient &client)
{
  int charcount=0;
  memset(linebuf,0,sizeof(linebuf));
  while (client.connected()) 
  {
    if (client.available()) 
    {
      char c = client.read();
      if (c>=' ' && charcount<sizeof(linebuf)-1)
      {
        linebuf[charcount]=c;
        charcount++;
      }
      else if (c == '\n') break; // end of line found, break while-loop
    }
  }
  return linebuf;
}


void setup(){
  Serial.begin(115200);
  Ethernet.begin(mac, ip);
  Serial.print("Server is at: ");Serial.println(Ethernet.localIP());
}

void loop()
{
  EthernetClient client;
  if (client.connect(server, 80)) 
  {
    int linecount=0;
    sendGetRequest(client);
    skipHeader(client);
    while (client.connected())
    {
      char* response= readClientLine(client);
      linecount++;
      if (linecount<10) Serial.print(' ');
      Serial.print(linecount);
      Serial.print(": ");
      Serial.println(response);
    }
    client.stop();
  }
  delay(60000); // pause 1 minute before next request
}

I ended up using the TextFinder code supplied by Zoomkat which worked flawlessly with only a few additional lines of code.

 if (client.available()) {
client.read();
 finder.find("l_fix");  // seek to the Results field
    long value = finder.getValue(); // get numeric value
    Serial.print(value);
    Serial.println(" last price");
  }

This simply outputs all the integers directly after the target string "l_fix"

The original works great for me though. I have been trying to give commands to my arduino through parsed HTTP GET requests. My only question is: What the hell is favicon.ico, and why does it keep echoing my requests?