ESP8266 HTTP(S) Response Time

Here’s the code again. No matter what I try there always seems to be a 3-4s delay after headers are received to receive the actual server response. Still can’t figure out why.

#include <ESP8266WiFi.h>
//WiFiClient client;

// Use this for HTTPS:
#include <WiFiClientSecure.h>
WiFiClientSecure client;

#include <ESP8266HTTPClient.h>

const char* ssid     = "BLAH";
const char* password = "BLAH";

//const int port = 80; // For HTTP
const int port = 443; // For HTTPS

void setup() {
  Serial.begin(115200);

  // Connect to WiFi
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Perform GET request to test server (dweet.io)
  GET();
}

void loop() {
  // Nothing here
}

void GET() {
  if (!client.connect("dweet.io", port)) { // Use dweet.io to test
    Serial.println("Connection failed");
  }
  client.setNoDelay(true);
  
  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.0\r\n") +
               "Host: dweet.io\r\n" + 
               "Connection: close\r\n\r\n");
  
  while (client.connected()) {
    String response = client.readStringUntil('\n');
    if (response == "\r") {
      Serial.println("<-- Headers received");
      break;
    }
  }
  String response = client.readStringUntil('\n');
  Serial.println(response);
}

//Using the HTTPClient library yields almost no delay
//void GET() {
//  HTTPClient http;
//  
//  http.begin("http://dweet.io/get/latest/dweet/for/esp8266test123456");
//  http.setReuse(true); // Keep-alive connection
//  int httpCode = http.GET();
//
//  if (httpCode > 0) {
//    String payload = http.getString();
//    Serial.println(payload);
//  }
//  
//  http.end();
//}

I already tried all that, and I posted the code in an earlier post.

No you didn’t:

  while (client.connected()) {
    String response = client.readStringUntil('\n');
    if (response == "\r") {
      Serial.println("<-- Headers received");
      break;
    }
  }
  String response = client.readStringUntil('\n');
  Serial.println(response);

You still don’t put anything out as long as there is a connection.

And you read only the first line of the response. Are you sure the server doesn’t provide more lines?

You still don't put anything out as long as there is a connection.

Sorry, maybe I'm not quite sure what you mean by this?

And you read only the first line of the response. Are you sure the server doesn't provide more lines?

I'm sure the server only provides one line because I get the proper response I'm expecting, but only after a 4s delay.

OK, I changed the code to print out everything as long as it's connected and I get the following response:

HTTP/1.1 200 OK
Access-Control-Allow-Origin: *
Content-Type: application/json
Content-Length: 62
Date: Tue, 27 Feb 2018 16:07:15 GMT
Connection: close

{"this":"failed","with":404,"because":"we couldn't find this"}

I just changed the while (client.connected()) loop to

while (client.connected()) {
    Serial.print(client.readString());
  }

Actually, everything printed out about 6s after it was connected to WiFi, so the headers were actually not received early. Any ideas why everything's being delayed?

Actually, everything printed out about 6s after it was connected to WiFi, so the headers were actually not received early. Any ideas why everything's being delayed?

You change the description in almost every post. Before you wrote that there is a delay from receiving the headers to the rest of the message. Now you have a delay from the setup of the WiFi to the result. Because that time period also includes the DHCP transfer, DNS lookup and in your case the SSL handshake there may be many things that may go wrong. Try to eliminate as much as possible (so no DNS lookup, no SSL handshake -> connect to IP address and use port 80 and no secure client).

Sorry for the confusion. From the original code I posted (before just printing everything) it printed "Headers received" even though, I guess, it really wasn't.

I tried using HTTP with port 80 and same thing happened. I will try direct IP address.

On another note, it looks like I should be using HTTP/1.1 since I used HTTP/1.0 but the headers returned HTTP/1.1. Is that correct?

I tried using the IP address instead, and still same results:

#include <ESP8266WiFi.h>
WiFiClient client;

// Use this for HTTPS:
//#include <WiFiClientSecure.h>
//WiFiClientSecure client;

#include <ESP8266HTTPClient.h>

const char* ssid     = "BLAH";
const char* password = "BLAH";

//const char* host = "dweet.io";
const char* host = "34.225.51.67";
const int port = 80; // For HTTP
//const int port = 443; // For HTTPS

void setup() {
  Serial.begin(115200);

  // Connect to WiFi
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Perform GET request to test server (dweet.io)
  GET();
}

void loop() {
  // Nothing here
}

void GET() {
  if (!client.connect(host, port)) { // Use dweet.io to test
    Serial.println("Connection failed");
  }
  client.setNoDelay(true);
  
//  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.0\r\n") +
//               "Host: dweet.io\r\n" + 
//               "Connection: close\r\n\r\n");

//  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.0\r\n") +
//               "Host: dweet.io\r\n\r\n");

  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.1\r\n") +
               "Host: " + host + "\r\n" + 
               "Connection: close\r\n\r\n");

//  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.1\r\n") +
//               "Host: 34.225.51.67\r\n\r\n");

  while (client.connected()) {
    Serial.print(client.readString());
  }
  
//  while (client.connected()) {
//    String response = client.readStringUntil('\n');
//    if (response == "\r") {
//      Serial.println("<-- Headers received");
//      break;
//    }
//  }
//  String response = client.readStringUntil('\n');
//  Serial.println(response);
}

On another note, it looks like I should be using HTTP/1.1 since I used HTTP/1.0 but the headers returned HTTP/1.1. Is that correct?

Theoretically the server should answer using HTTP 1.0 if the request was done using HTTP 1.0, so this is a failure on the server side, I don't think you should adapt because of that.

I tried using the IP address instead, and still same results:

const char* host = "34.225.51.67";
  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.1\r\n") +
               "Host: " + host + "\r\n" +
               "Connection: close\r\n\r\n");

That inserts the IP address in the Host header which might lead to using the wrong configuration on the server.

Insert a serial output after the client.connect() call to have a timing of the different stages of the request. Do the same after client.print() with the request header. Then check the timing on the serial monitor. Alternatively you can calculate the timing on the ESP and print the used time so you have exact values. Post the results.

Just to verify that: Your ESP is connected to the same WiFi network as your controlling PC, isn't it?

OK I'll stick with HTTP/1.0 for now. I also just used "Host: dweet.io" instead of IP address for the client.print(). Here's the new client.print() I used:

client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.0\r\n") +
               "Host: dweet.io\r\n" + 
               "Connection: close\r\n\r\n");

And yes, I'm using the same WiFi network as on my PC. Basically I'm expecting a quick reply, just as if I load "http://dweet.io/get/latest/dweet/for/esp8266test123456" in my browser.

I added the millis() time at various places and here are the results:

  • Time before connecting to server: 7392
  • After connecting to server: 7415
  • After client.print(): 7415
  • After the while(client.connected()) loop: 13249

So it's taking about 5.8s from the client.print() to receiving the response from the server.

Can you please try that again but insert a printout of the millis() value after the Serial.print() inside the while loop? I have a guess about the reason for your problems and that output may help to find the correct code part.

I inserted a millis() line after the Serial.print like this:

while (client.connected()) {
  Serial.print(client.readString());
  Serial.print("loop: "); Serial.println(millis());
}

and it gave out 13643 whereas the time right after the entire while loop was 13644.

Is there a question?

GrooveFlotilla:
Is there a question?

Yes, there very much is a question here. I'm trying to figure out why there is such a high latency (6s or so) for getting a server response via HTTP using the ESP8266 WiFiClient or WiFiClientSecure libraries as opposed to using the HTTPClient library which yields near-instant responses.

Sorry, I came in on the third page & didn't notice the earlier pages.

The problem is readString(). Look at the implementation:

String Stream::readString()
{
  String ret;
  int c = timedRead();
  while (c >= 0)
  {
    ret += (char)c;
    c = timedRead();
  }
  return ret;
}

As you can see it reads a C-string (null byte terminated) from the stream but the HTTP protocol sends no zeros.
So it waits for the zero until the timeout occurs. It might be necessary to parse the output yourself and don't use methods that are developed for other purposes.

Hmmm that's interesting because the HTTPClient library uses readStringUntil(). How would I code it myself so that there isn't a delay? Should I just use client.read() and parse the bytes together?

pylon:
The problem is readString(). Look at the implementation:

String Stream::readString()

{
 String ret;
 int c = timedRead();
 while (c >= 0)
 {
   ret += (char)c;
   c = timedRead();
 }
 return ret;
}




As you can see it reads a C-string (null byte terminated) from the stream but the HTTP protocol sends no zeros.
So it waits for the zero until the timeout occurs. It might be necessary to parse the output yourself and don't use methods that are developed for other purposes.

There's no C-string here. timedRead() reads one character from the stream and appends it to ret (a String) if it's not negative. timedRead() returns -1 if there's a timeout while trying to read from the stream.

Thanks, guys, finally figured it out! With this code there's no delay. It was that darn simple:

while (client.connected()) {
    if (client.available()) {
      char c = client.read();
      Serial.print(c);
    }
  }
  Serial.println();
1 Like

There's no C-string here. timedRead() reads one character from the stream and appends it to ret (a String) if it's not negative. timedRead() returns -1 if there's a timeout while trying to read from the stream.

It reads a C-type string, it doesn't read it into a C-string. I hope you see that this is something different.

Hmmm that's interesting because the HTTPClient library uses readStringUntil().

readStringUntil() is something different. If called with "\n" parameter it reads line after line.

pylon:
It reads a C-type string, it doesn't read it into a C-string. I hope you see that this is something different.

That's still not true. It reads from a Stream, not a C-string. The Stream in this case is a WiFiClient. Stream.read() (and by extension, Stream.timedRead()) doesn't look for a null terminator.