ESP8266 HTTP(S) Response Time

Hi there,

With the ESP8266HTTPClient library using “http.begin()” and “http.GET()” functions the GET requests I am able to read the server response almost immediately using “payload = http.getString()”. However, when using the manual way of connecting with WiFiClient or WiFiClientSecure and sending and receiving the lines manually, it takes up to 5s to read the server response. I receive the headers almost immediately after sending, but the actual response content comes 3-4s later. There’s nothing special I’m doing. I just use “client.connect()” then use a single “client.print()” line to do a GET request and then use “client.readStringUntil(’\n’)” to read the response. Pretty straightforward and it all works, but the server response are slow to arrive.

Here’s the code using the ESP8266HTTPClient library:

HTTPClient http;
http.begin("http://server.io/blah?foo=bar");
int httpCode = http.GET();

if (httpCode > 0) {
  String payload = http.getString(); //Get the request response payload
  Serial.println(payload);
}

http.end();

Here’s the code going it “manually”:

WiFiClient client;
if (!client.connect("server.io", 80)) {
  Serial.println("Connection failed");
}

client.print("GET /blah?foo=bar HTTP/1.1\r\n\r\n");

while (client.connected()) {
  String response = client.readStringUntil('\n');
  if (response == "\r") {
    Serial.println("<-- Headers received");
    break;
  }
}
String response = client.readStringUntil('\n');
Serial.println(response);

What’s the difference here and why is it taking so long doing it manually compared to the HTTPClient functions? Is there something hidden behind the library that I’m missing when doing it manually? I get the same 4-5s delayed response time with WiFiClientSecure as well.

Thanks!

client.print("GET /blah?foo=bar HTTP/1.1\r\n\r\n");

That doesn't encode a correct GET request for HTTP 1.1. It's not always a good idea to use the newest version number if you don't know what exactly you're doing. I don't know what server is responding at your "server.io" but you probably should use HTTP 1.0 and add a "Host: server.io" header line. Depending on the server you may need additional information.

Thanks for the comment. I HTTP/1.0 with the "Host: server.io" line with "Connection: keep-alive" as well but that didn't help. It's still taking about 3-4s to get a response each time. But again, it's receiving the headers right away, that's what gets me.

I HTTP/1.0 with the "Host: server.io" line with "Connection: keep-alive" as well but that didn't help

That should be "Connection: close". Otherwise the client has to parse the header and read the correct number of bytes.

But I want to read multiple times, so if I close the connection I’ll have to re-connect. I also tried to close the connection but it’s still delayed. As far as I know HTTP/1.1 has keep-alive by default, which is why I leave it out with 1.1 but explicitly use it for 1.0.

But I want to read multiple times, so if I close the connection I'll have to re-connect. I also tried to close the connection but it's still delayed. As far as I know HTTP/1.1 has keep-alive by default, which is why I leave it out with 1.1 but explicitly use it for 1.0.

Your code checks for the connection to close so the connection should close.

Post complete code, maybe the error is somewhere else.

You can check with your PC using a telnet client. Just type in the same lines as you have in your code. If the delay is also in the telnet client, then the server reacts like this with the given header lines and you might to adapt them to change the server's behavior.
If you have no delay in the telnet client on the PC, the error is probably within the code part that you're hiding from us.

The code is nothing special:

#include <ESP8266WiFi.h>
WiFiClient client;

// Use this for HTTPS:
//#include <WiFiClientSecure.h>
//WiFiClientSecure client;

const char* ssid     = "BLAH";
const char* password = "BLAH";

const int port = 80; // For HTTP
//const int port = 443; // For HTTPS

void setup() {
  Serial.begin(115200);

  // Connect to WiFi
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  // Perform GET request to test server (dweet.io)
  GET();
}

void loop() {
  // Nothing here
}

void GET() {
  if (!client.connect("dweet.io", port)) { // Use dweet.io to test
    Serial.println("Connection failed");
  }  
  client.print(String("GET /get/latest/dweet/for/esp8266test123456 HTTP/1.1\r\n") +
               "Host: dweet.io\r\n\r\n");
  
  while (client.connected()) {
    String response = client.readStringUntil('\n');
    if (response == "\r") {
      Serial.println("<-- Headers received");
      break;
    }
  }
  String response = client.readStringUntil('\n');
  Serial.println(response);
}

Same results with HTTPS and with Amazon AWS as well and I also tried with “Connection: close” and still the same thing. However, using the HTTPClient library the response is near immediate. Do you think it has to do with some header stuff I’m not sending?

Thanks!

Do you think it has to do with some header stuff I'm not sending?

Yes. In the current version it'll wait until the timeout for the keep-alive is reached because your code still waits for the connection to close, it won't print any response before that.

Using "Connection: close" it does the same thing.

This really doesn't make sense to me. The HTTPClient library sends the following headers:

  • HTTP1.0 or HTTP1.1 (1.0 by default)
  • Host: blah.io
  • User-Agent: whateverYouWant
  • Connection: close (default)
  • If HTTP1.0 is not used then it also sends "Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0"

I tried sending these exact headers, still no luck; I experience the same weird 4s delay after receiving the headers.

Am I overlooking something here? There has to be something hidden in the HTTPClient library that I'm missing.

I tried sending these exact headers, still no luck; I experience the same weird 4s delay after receiving the headers.

Have you tried on your PC with the telnet client? Send exactly the same header lines as your sketch would do. If you notice the same delays, it's something in the header lines. If you get immediate response, there must be an error in your sketch.

For some reason using telnet on Windows 10 command prompt didn't work for me. It simply doesn't connect to to the server. I tried using the command "open dweet.io 443" and it said "Connecting to dweet.io..." for a long time then said connection failed.

Any ideas?

I tried

telnet dweet.io 80

and got a connection immediately (I use port 80 because you do in your sketch too). I guess you should check your network connectivity. It seems there is a problem not Arduino related.

I just tried the command you wrote above in command prompt and the cursor just blinks for a while. Does that mean it's connected? Aren't you supposed to use the "open" command? How do I send the other commands?

And no, it's not a connection problem here because the HTTPClient library works fine, and so does the "manual" attempt with the WiFiClient or WiFiClientSecure libraries. Again, they all work, but the manual methods results in a 4s delay after headers are received.

I just tried the command you wrote above in command prompt and the cursor just blinks for a while. Does that mean it's connected? Aren't you supposed to use the "open" command? How do I send the other commands?

I never use Windows, so I cannot tell you how exactly it should look. On my box it shows:

telnet dweet.io 80
Trying 34.231.80.209...
Connected to dweet.io.
Escape character is '^]'.

There's definitely no "open" command as the telnet command is there to open a connection.
What kind of "other commands" do you expect to have? That's just a telnet connection, more or less a simple TCP connection. The available commands depend on what service is listening on the connected port.

As soon as I have the connection open I can type the request header lines to the web server and it will answer. But we the header lines of your sketch it waits about 10s as it defaults to keep-alive and your sketch doesn't recognize the answer until the connection is closed. Maybe you should fix that or change to HTTP 1.0.

Attached is a picture of the telnet commands available for Windows 10 command prompt.

In Arduino I tried HTTP 1.0 and “Connection: close” and neither had any effect on the response time.
I’m pretty stumped right now…

Capture.PNG

Have you tried

open dweet.io 80

According to the posted documentation this should work. Sad that Microsoft got lost of their real telnet client. If that's the only thing they provide to customers anymore I know again why I use another operating system.

Yes, that's what I was using but it just said "Connecting to dweet.io..." for a long time then timed out. Also tried port 443.

androidfanboy:
Yes, that's what I was using but it just said "Connecting to dweet.io..." for a long time then timed out. Also tried port 443.

That means you either have some weird firewall setup or you have sporadic connectivity problems.

Please implement all my recommendations (HTTP/1.0, No keep-alive, Host: header and no wait for closing the connection) and try again. If it doesn't work, post the sketch you tested with.

I already tried all that, and I posted the code in an earlier post. Again, it's not that it doesn't "work", but that there's a 4s delay before receiving the server response after receiving the headers.