Uno+WiFi Shield: sometimes not able to read returned HTTP data from client

Hello all,

I am having a problem with the official Arduino WiFi shield where I am unable to read the data returned from a GET request. The problem only occurs when the requests are made in quick succession...

If a GET request is made every 10 seconds, I am able to read the returned status code. However, if the requests are made very quickly, I get a response of "." in the terminal. I think this is a problem with the baud rate of the shield. My serial terminal is running at 115200 baud rate. I'm not too sure what the rate of the shield is set to or how to go about changing it.

Here is my code:

void loop() {

  client.flush();
  Serial.flush();

  do {
    printAndPing();
  } while (!timeToPush);

  getRequest(ID);

}


void listenComm() {
  char charRead = NULL;
  unsigned char statusCount=0;
  while (client.available() && statusCount<12) {
    charRead = client.read();
    Serial.print(charRead);
    statusCount++;
  }
  client.flush();
}


void getRequest(unsigned long ID) {
  radio.stopListening();
  Serial.println("Time to push!");
  timeToPush = false;

  unsigned long user = ID / DIVISOR;
  unsigned long total = ID % DIVISOR;

  // if there's no net connection, but there was one last time
  // through the loop, then stop the client:
  //if (!client.connected() && lastConnected) {
  //  Serial.println("Closing the previous connection now...");
    client.stop();
  //}

  if (client.connect(server, 80)) {
    lastConnected = true;
    client.print("GET ...");
    /* GET REQUEST OMITTED FOR SECURITY *
     * Please trust me that the request *
     * itself isnt wrong :P             */
    client.println("Connection: close");
    client.println();
  }
  else {
    lastConnected = false;
    // if couldn't make a connection:
    if (connectFailCount<3) {
      connectFailCount++;
      Serial.print("ERROR: connection failure #");
      Serial.println(connectFailCount);
      client.stop();
    }
    // if failed 5 total times, reset:
    else {
      connectFailCount = 0;
      Serial.print("ERROR: connection failure CRITICAL!");
      Serial.println("Disconnecting...");
      Serial.println("Performing connection reset.");
      Serial.println("----------------------------");
      client.stop();
      WiFi.disconnect();
      delay(1500);
      connectToTheInternet();
      delay(3000);
    }
  }
  radio.startListening();
  listenComm();
}

void connectToTheInternet() {
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);
  }
  Serial.println("Connected to the interwebs!");
  printWifiStatus();
}

// RF Code for radio frequency input:
void printAndPing() {
  if ( radio.available() ) {
    // Dump the payloads until we've gotten everything
    radioDone = false;
    while (!radioDone)
    {
      Serial.println();
      Serial.print("Just received signal from device #: ");
      // Fetch the payload, and see if this was the last one.
      radioDone = radio.read( &ID, sizeof(unsigned long) );
      Serial.println(ID);
      timeToPush = true;
    }
  }
}

Here is a sample output:

Just received signal from device #: 100002371
Time to push!
HTTP/1.1 422
Just received signal from device #: 100001280
Time to push!

Just received signal from device #: 100000367
Time to push!
HTTP/1.1 422
Just received signal from device #: 100002371
Time to push!
HTTP/1.1 422
Just received signal from device #: 100000367
Time to push!
HTTP/1.1 422
Just received signal from device #: 100002371
Time to push!
HTTP/1.1 422
Just received signal from device #: 100001280
Time to push!

Just received signal from device #: 100000367
Time to push!
HTTP/1.1 422
Just received signal from device #: 100002371
Time to push!
HTTP/1.1 422
Just received signal from device #: 100001280
Time to push!

Just received signal from device #: 100000367
Time to push!
HTTP/1.1 422

  char statusHTTP[12];
  char charRead = NULL;
  unsigned char statusCount=0;

  while (client.available() && statusCount<12) {
    charRead = client.read();
    Serial.print(charRead);
    statusCount++;
  }

What is statusHTTP[] for? It's not used anywhere.

Why are you not storing the values read?

  // if there's no net connection, but there was one last time
  // through the loop, then stop the client:
  //if (!client.connected() && lastConnected) {
  //  Serial.println("Closing the previous connection now...");
    client.stop();
  //}

Everything but the client.stop() is commented out. Why? Why isn't client.stop() also commented out?

Here is my code:

You mean "here is some of my code that has nothing to do with the problem I am having".

Some of the OP's problem is here.

  if (client.connect(server, 80)) {
    lastConnected = true;
    client.print("GET ...");
        {... GET procedure here...}
    client.println("Connection: close");
    client.println();
  }

The server responds with http error 422. You can Google search for that and get part of the answer.

PaulS:
What is statusHTTP[] for? It's not used anywhere.
Why are you not storing the values read?

Because I don't need to store them; therefore I don't store them. and statusHTTP[] is an array that is used for something that isnt implemented yet.

PaulS:
Everything but the client.stop() is commented out. Why? Why isn't client.stop() also commented out?

Because I want the client to stop after every connection otherwise sometimes connections overlap and cause malformed requests. Otherwise, it would be running a logical check that would increase the amount of work the processor has to do per every execution.

PaulS:
You mean "here is some of my code that has nothing to do with the problem I am having".

EDIT: I apologize for saying you were being a dick. But code I did post initially has everything to do with the problem I'm having, albeit it might not be complete. I updated the original post with all code.

The problem here is the responses not coming back to the Arduino. there are 0 problems with making the actual get requests since I see all of the requests on my server.

SurferTim:
Some of the OP's problem is here.
The server responds with http error 422. You can Google search for that and get part of the answer.

Hey, thanks for the reply. But the 422 error doesnt matter. I am running the server, and I know the status code is returned as a 422. Thats because its making a request thats actually forbidden by a database rule I wrote for that page. So thats actually a good thing. And just to clarify, I omitted the actual address of the GET request from my post because I am not allowed to share the URL of the page I am working on due to security concerns and company policy.

The actual problem is that SOME of the requests return a 422 while others do not return anything while still showing up on the server logs.

Just a thought:

void loop() {
  // If you have received a packet here, what will the next instruction do?
  client.flush();

  Serial.flush();
  do {
    printAndPing();
  } while (!timeToPush);
  getRequest(ID);
}

edit: BTW, you may think PaulS is a dick, a troll, or a member of the "Chuck Norris" club, but he is correct. If you do not post all your code, then you fall under one of my rules also:
"If the user thinks it is not important enough to include, it probably is".
Where is ALL your code?

If you think that is too much to ask, and think the same of me, then I prefer Mr. A##hole, if you please!

SurferTim:
Just a thought:

Right, which is what I thought might be messing it up? But I have the client.read() function right at the end of the getRequest() function. Do I need to be accounting for the delay of getting the responses? Ill give it a shot right now and let you know. Thanks for the suggestion!

So for reference, here is a COMPLETE walkthrough of my main loop() without any external function references:

void loop() {

  client.flush();
  Serial.flush();

  /* ---------------------------------------- */
  /*          LISTEN TO RADIO SIGNAL          */
  /* ---------------------------------------- */
  /* do...while loop from "void loop() {...}" */

  do {
    if ( radio.available() ) {
      // Dump the payloads until we've gotten everything
      radioDone = false;
      while (!radioDone) {
        Serial.println();
        Serial.print("Just received signal from device #: ");
        // Fetch the payload, and see if this was the last one.
        radioDone = radio.read( &ID, sizeof(unsigned long) );
        Serial.println(ID);
        timeToPush = true;
      }
    }
  } while (!timeToPush);


  /* ---------------------------------------- */
  /* RADIO SIGNAL RECEIVED, ISSUE GET REQUEST */
  /* ---------------------------------------- */
  /* getRequest(); Function shown here:       */

  radio.stopListening();  /* Stop radio frequency communication
                             once radio is pinged.           */
  timeToPush = false;     // Reset once we are out of do...while
  Serial.println("Time to push!");


  unsigned long user = ID / DIVISOR;
  unsigned long total = ID % DIVISOR;

  client.stop();

  bool connected = client.connect(server, 80);
  if (connected) {
    client.print("GET /swipes?auth_token=");
    /* GET REQUEST OMITTED FOR SECURITY *
     * Please trust me that the request *
     * itself isnt wrong :P             */
    client.println("Connection: close");
    client.println();
  }
  else {  // == if(!connected) == if (!client.connect(server,80))
    lastConnected = false;
    // if couldn't make a connection:
    if (connectFailCount<3) {
      connectFailCount++;
      Serial.print("ERROR: connection failure #");
      Serial.println(connectFailCount);
      client.stop();
    }
    // if connection failed 5 total times, reset:
    else {
      connectFailCount = 0;
      Serial.print  ("ERROR: connection failure CRITICAL!");
      Serial.println("Performing connection reset.");
      Serial.println("-----------------------------------");
      client.stop();
      WiFi.disconnect();
      delay(1500);
      // connectToTheInternet(); Function shown here:
        while ( status != WL_CONNECTED) {
          Serial.print("Attempting to connect to SSID: ");
          Serial.println(ssid);
          status = WiFi.begin(ssid, pass); // Connect to WPA/WPA2 network
        }
        Serial.println("Connected to the interwebs!");
        printWifiStatus();
      delay(3000);
    }
  }
  delay(1000);


  /* ---------------------------------------- */
  /* GET REQUEST COMPLETED, LISTEN TO CLIENT  */
  /* ---------------------------------------- */
  /* listenComm(); Function shown here:       */

  statusCount=0;
  while (client.available() && statusCount<12) {
    charRead = client.read();
    Serial.print(charRead);
    statusCount++;
  }
  while (client.available()) client.read();
  Serial.println("\n---------------------------------------------");

  /* ---------------------------------------- */
  /*        END OF MAIN "VOID LOOP()"         */
  /* ---------------------------------------- */

}

SurferTim:
edit: BTW, you may think PaulS is a dick, a troll, or a member of the "Chuck Norris" club, but he is correct. If you do not post all your code, then you fall under one of my rules also:
"If the user thinks it is not important enough to include, it probably is".
Where is ALL your code?
If you think that is too much to ask, and think the same of me, then I prefer Mr. A##hole, if you please!

You're right. And my apologizes to @PaulS. It was just the way he phrased it that pissed me off. It's been a long few days for me. But I was pretty sure I included all of the relevant code. I did miss putting in the main loop so my apologies. All the main loop does is issue GET requests based on RF signal inputs.

EDIT: I updated my original post to include all functions.

Maybe it is just me, but where are the includes and setup function? But despite that, you MUST wait for the server to close the connection. That is the signal it is finished. Once you are connected to the server, use this:

while(client.connected()) {
    while (client.available()) {
      charRead = client.read();
      Serial.print(charRead);
    }
}
client.stop();

If the connection breaks (hardware fail enroute), you must include a timeout in that routine, but we will discuss that later.

SurferTim:
Maybe it is just me, but where are the includes and setup function?

Is the setup function really necessary? Like I said, every single GET request goes through to the server. It's just that some of the responses arent outputted to the serial terminal. However, every request gets a response from the server.

/* Standard Includes */
#include <string.h>
#include <stdlib.h>
/* WiFi library */
#include <WiFi.h>
#include "Credentials.h"
/* EEPROM Wire library */
#include <Wire.h>
#include <Debug.h>
/* MiRF Chip SPI library */
#include <SPI.h>
#include <RF24.h>
#include <nRF24L01.h>
#include <MirfHardwareSpiDriver.h>
/* Program Variable Library */
#include "hub.h"

void setup() {
  Serial.begin(BAUD_SERIAL);
  Wire.begin();
  radio.begin();

  checkShield();
  Serial.println("Hello World!\n");
  connectToTheInternet();
  delayFirstRun();

  radio.openWritingPipe(pipes[1]);
  radio.openReadingPipe(1,pipes[0]);
  radio.startListening();
}

void checkShield() {
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    // don't continue:
    while(true);
  }
}

void delayFirstRun(){
  Serial.print("STARTING RADIO IN ");
  for (int i=0; i<5; i++){
    delay(1000);
    Serial.print(5-i);
    Serial.print("... ");
  }
  Serial.println();
}

void connectToTheInternet() {
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    status = WiFi.begin(ssid, pass);
  }
  Serial.println("Connected");
  printWifiStatus();
}

SurferTim:
But despite that, you MUST wait for the server to close the connection. That is the signal it is finished. Once you are connected to the server, use this:

while(client.connected()) {

while (client.available()) {
      charRead = client.read();
      Serial.print(charRead);
    }
}
client.stop();



If the connection breaks (hardware fail enroute), you must include a timeout in that routine, but we will discuss that later.

My procedure does the same thing. The client.stop() isnt issued until the next getRequest() function is started. But I tried your suggestion as well, and it affect the problem. It actually made it worse, and none of the getRequests get a response.

Good guess, but no cigar.

  while (client.available() && statusCount<12) {
    charRead = client.read();
    Serial.print(charRead);
    statusCount++;
  }
  client.flush();

That is not even close to the code I recommended. Where is the while(client.connected()) check?

SurferTim:
Good guess, but no cigar.
That is not even close to the code I recommended. Where is the while(client.connected()) check?

Touche sir. That works.

Touche sir. That works.

That is what I like to hear! :slight_smile:

SurferTim:

Touche sir. That works.

That is what I like to hear! :slight_smile:

Thank you so much for your continued help.
If its alright, id like to ask you a question about the while(client.connected) line. So now there is a significant delay between the get request being issued and the response being read. This device that I am working on is going to have to make a GET request every couple seconds. Is there a way to cut that delay down? Would it help to build the entire GET request as a string before sending it to the client?

I promised you a hardware fail abort routine. I try not to go back on my word. Here is a 10 second timeout routine to apply to the code you already have. That should be plenty of time for a server to respond.

edit: Server and network traffic will determine the lag in response. The less traffic or faster server/connection, the less response delay.

SurferTim:
I promised you a hardware fail abort routine. I try not to go back on my word. Here is a 10 second timeout routine to apply to the code you already have. That should be plenty of time for a server to respond.
Reconnecting to Ethernet - #7 by SurferTim - Networking, Protocols, and Devices - Arduino Forum

edit: Server and network traffic will determine the lag in response. The less traffic or faster server/connection, the less response delay.

I added that timeout in there. At this point its server side optimization i guess. Thank you so much again for your help.