Problems with WiFi and HTTPclient

Projects Networking, Protocols, and Devices
1

Hi guys,

First post...

I am using an ESP8266 device to connect to a Tapo plug device using WiFi.

I have a small project written in python that works fine on a PC. I can drive a couple of Tapo plugs via this API fine (someone has cracked the protocol). I have looked at MicroPython for the Arduino, but I can't find all of the necessary encryption, decryption and hashing algorithms to port this over from the PC to MicroPython.

I have found all the libraries I think I need with the Arduino cloud-based IDE and I am working on converting the code over. I have hit a little snag that has stalled me for a couple of days, and I wonder if someone can point me in the right direction?

I am using the following WiFi libraries:

#include <ESP8266WiFi.h> #include <ESP8266HTTPClient.h> #include <WiFiClient.h>

I am creating an instance of the WiFiClient (client) and the HTTPClient (http).

I am trying to communicate with the Tapo plug on IP address "192.168.0.42" using http: and port 80.

I am using the call:

if( !http.begin( client, IP, 80, "/app/handshake1", false ) ) { /* ERROR */ }

This call works ok. I am able to POST a byte array to the Tapo plug and I get a 200 response code and a byte array back in return. I am performing the checks on the returned byte array and they indicate that my Arduino end and the Tapo plug have authenticated themselves. So far so good.

Now my problem...

I now need to communicate with the same host but a different uri as follows:

if( !http.begin( client, IP, 80, "/app/handshake2", false ) ) { /* ERROR */ }

Again, this call works ok.

I now POST some data to it (this data has been 'checked' as correct in the previous 'authentication' code).

However, I receive a response code of 400 from the POST.

I have instrumented up the Python code, and the equivalent POST returns a response code of 200, so something is going wrong. Now, it could be that the byte array I am sending in the POST is malformed in some way. Unfortunately, I can't compare apples with apples due to the Tapo plug and my application using random data bytes to obfuscate the communications. However, I have checked the best that I can that my code is correct...

On thing I was concerned about was the closing of the TCP/IP connection to the plug between POSTs (i.e. between calling http.begin( ... )).

This is where I am having fun...

When I created the http instance, I invoked http.setReuse( true ) - although this appears to be the default when the HTTPClient interface is created.

I have been trying to follow the source code through for the HTTPClient, but this is where I am having problems:

I found some class documentation here: ESP8266: HTTPClient Class Reference. But this seems out of date.

I found some source code here: Arduino/libraries/ESP8266HTTPClient at master · esp8266/Arduino · GitHub. But this doesn't seem to be what the Arduino IDE compiler is using. For example, there is a ".setURL" method, but when I try to use this, the compiler tells me that no such method exists.

I have been specifically trying to follow the "begin" method using the parameters I am passing. My understanding of the code is that if the host (in my case the IP address) has not changed, then the connection will remain open. However, there are a _reuse and a _canReuse items embedded in the logic. But, as I have mentioned above, there is no guarantee that the source code I have found and what is being used by the Arduino compilation system are the same.

I have seen mention about the use of the setReuse method - but in the context of the connection being reused when it should not...

Any ideas anyone?

Dave

2

First read the pinned post re 'How to get the most from the forum'
Second, if you think libraries that have been used millions of times are the source of your problem, then you might need to go to a different forum.

3

HTTPClient::setUrl(String) was added in version 2.5.1. The main page of the repo indicates the Latest release version in the right-hand column: 3.1.2. So the first thing to check is which version of the ESP8266 board you have installed.

If you get an unhappy response status like 400, the body of the response may have more specifics about what is wrong; print it.

4

Thanks for the suggestions.

I set 'verbose' mode yesterday - but that didn't seem to do anything (with the compilation output) until I logged in today. Now I get plenty of output to review!

The board level revision is 2.5.0 - so this does confirm that setURL (if added into 2.5.1) is missing.

The library manager indicates the ESP8266HTTPClient library is Version 1.2 (which does match with what is in GitHub).

I added a check after the unhappy post - but the http.getSize() method returned 0, so it appears there is no additional information passed back from the Tapo plug.

The diagnostic text output from DEBUG HTTP_CLIENT (I am using the cloud IDE) still seems to differ from the source code for ESP8266HTTPClient.cpp on GitHub. Which is confusing if it is version 1.2?

The diagnostic output I am observing is consistent with the TCP port being closed and re-opened despite what I have specified.

For example, I am getting the debug message [HTTP-Client][end] tcp stop rather than [HTTP-Client][end] tcp keep open for reuse implying that the expression _reuse && _canReuse is returning false rather than true within method void HTTPClient::disconnect(bool preserveClient) after retrieving the data from the prior POST that successfully completes.

Diagnostic output from the POST that worked:

`
[HTTP-Client] connected to 192.168.0.18:80
[HTTP-Client] sending request header

POST /app/handshake1 HTTP/1.1
Host: 192.168.0.18
User-Agent: ESP8266HTTPClient
Connection: keep-alive
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Content-Length: 16

[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'Set-Cookie: TP_SESSIONID=DC5B11FD0FA99FDD7D409FDA947998DA;TIMEOUT=86400'
[HTTP-Client][handleHeaderResponse] RX: 'Server: SHIP 2.0'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 48'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/html'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 48
httpResponseCode (handshake1)=200
http.getSize()=48
[HTTP-Client][writeToStreamDataBlock] connection closed or file end (written: 48).
[HTTP-Client][end] tcp stop
`

Followed by the POST that failed:

`
[HTTP-Client] connected to 192.168.0.18:80
[HTTP-Client] sending request header

POST /app/handshake2 HTTP/1.1
Host: 192.168.0.18
User-Agent: ESP8266HTTPClient
Connection: keep-alive
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Content-Length: 32

[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 400 Bad Request'
[HTTP-Client][handleHeaderResponse] RX: 'Server: SHIP 2.0'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 0'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/html'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 400
httpResponseCode (handshake2)=400
The httpResponseCode (handshake2) returned an error condition.
http.getSize()=0
`

Notice the Connection: keep-alive but the message [HTTP-Client][end] tcp stop at the end of the 'good' POST.

Also, the second POST output [HTTP-Client] connected to 192.168.0.18:80 should not be present but [HTTP-Client] connect: already connected, reusing connection should be (I think)...

I am sort of thinking here that the library code for 2.5.0 may be flawed?

Is there any way to gain access to the source code for specific libraries held within the cloud IDE, or can I load the latest HTTPClient library from GitHub?

Dave

5

Incidentally, this is the initial debug output:

FQBN: esp8266:esp8266:nodemcuv2:dbg=Serial,lvl=HTTP_CLIENT
Using board 'nodemcuv2' from platform in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0
Using core 'esp8266' from platform in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0
6

Which one is that exactly? If there are multiple matches for the same header, the build process employs an algorithm to pick one consistently. It may pick the one from the board over any add-ons. At the end of a verbose compile, it will list all libraries used (including those from the board); so look for something like

Using library ESP8266HTTPClient at version 2.5.0 in folder

The folder location will either be under Arduino15/packages for the board or {your-sketchbook}/libraries for an add-on.

If you're using version 2.5.0, then its reuse logic is broken. Searching for any usage of _canReuse, the only one of consequence

                if(headerName.equalsIgnoreCase("Connection")) {
                    _canReuse = headerValue.equalsIgnoreCase("keep-alive");
                }

requires keep-alive from a response header. That's invalid in a few ways. The current version 3.1.2 correctly

                constexpr auto httpVersionIdx = sizeof "HTTP/1." - 1;
                _canReuse = _canReuse && (headerLine[httpVersionIdx] != '0');
                _returnCode = headerLine.substring(httpVersionIdx + 2, headerLine.indexOf(' ', httpVersionIdx + 2)).toInt();
                _canReuse = _canReuse && (_returnCode > 0) && (_returnCode < 500);

assumes reuse with HTTP 1.1 (if the response is not a server error) unless

                if(_canReuse && headerName.equalsIgnoreCase(F("Connection"))) {
                    if (headerValue.indexOf(F("close")) >= 0 &&
                            headerValue.indexOf(F("keep-alive")) < 0) {
                        _canReuse = false;
                    }
                }

it's Connection: close

Having said all that, the 400 you're getting may be unrelated. As you said, it could be the byte array.

7

I was looking for those lines the other day and couldn't find them. I must have been going blind, because I found them today...

Using library ESP8266HTTPClient at version 1.2 in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0/libraries/ESP8266HTTPClient 
Using library ESP8266WiFi at version 1.0 in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0/libraries/ESP8266WiFi 
Using library Adafruit SSD1306 at version 2.5.15 in folder: /run/arduino/directories-data/internal/Adafruit_SSD1306_2.5.15_f1dc1246f6c87beb 
Using library Adafruit GFX Library at version 1.10.7 in folder: /run/arduino/custom-libraries/Adafruit GFX Library 
Using library SPI at version 1.0 in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0/libraries/SPI 
Using library Wire at version 1.0 in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0/libraries/Wire 
Using library Crypto at version 0.4.0 in folder: /run/arduino/custom-libraries/Crypto 
Using library ESP8266 Microgear at version 1.2.4 in folder: /run/arduino/directories-data/internal/ESP8266_Microgear_1.2.4_afede7f269a7543b 
Using library Adafruit BusIO at version 1.17.2 in folder: /run/arduino/directories-data/internal/Adafruit_BusIO_1.17.2_a44a5fe9533d10ba 
Using library EEPROM at version 1.0 in folder: /run/arduino/directories-data/packages/esp8266/hardware/esp8266/2.5.0/libraries/EEPROM

Anyhow, here they are.

I will have another look at my application code tomorrow.

Dave

8

I just had a look at the library for ESP8266HTTPClient and found I could edit the source code for the library. I didn't of course, but I used this to cross-check the source code repository on GitHub for the latest. Bizarrely, the file headers have not been updated with the latest dates, modifications etc. I would have expected (as a minimum) for someone to have updated the file headers and incorporated at least the date of the modification(s).

So, I suppose my next quest is how to incorporate the latest ESP8266HTTPClient .h and .cpp files into my project?

Dave

9

Ah, here's the problem. Adding some judicious word wrap to line up the version numbers:

To compare, these are the library versions included with the esp32 board, version 3.0.7

$ pwd
/home/kenb4/.arduino15/packages/esp32/hardware/esp32/3.0.7/libraries
$ rg version --glob=library.properties
SPI/library.properties
2:version=3.0.7

Update/library.properties
2:version=3.0.7

SD/library.properties
2:version=3.0.7

SPIFFS/library.properties
2:version=3.0.7

WiFiProv/library.properties
2:version=3.0.7

TFLiteMicro/library.properties
2:version=3.0.7

EEPROM/library.properties
2:version=3.0.7

HTTPUpdate/library.properties
2:version=3.0.7

PPP/library.properties
2:version=3.0.7

FFat/library.properties
2:version=3.0.7

Ticker/library.properties
2:version=3.0.7

RainMaker/library.properties
2:version=3.0.7

SD_MMC/library.properties
2:version=3.0.7

Network/library.properties
2:version=3.0.7

FS/library.properties
2:version=3.0.7

HTTPUpdateServer/library.properties
2:version=3.0.7

Zigbee/library.properties
2:version=3.0.7

ESP_SR/library.properties
2:version=3.0.7

SimpleBLE/library.properties
2:version=3.0.7

Wire/library.properties
2:version=3.0.7

NetBIOS/library.properties
2:version=3.0.7

Ethernet/library.properties
2:version=3.0.7

HTTPClient/library.properties
2:version=3.0.7

WiFi/library.properties
2:version=3.0.7

LittleFS/library.properties
2:version=3.0.7

WebServer/library.properties
2:version=3.0.7

BLE/library.properties
2:version=3.0.7

Insights/library.properties
2:version=3.0.7

USB/library.properties
2:version=3.0.7

AsyncUDP/library.properties
2:version=3.0.7

ArduinoOTA/library.properties
2:version=3.0.7

ESP_I2S/library.properties
2:version=3.0.7

DNSServer/library.properties
2:version=3.0.7

ESP_NOW/library.properties
2:version=3.0.7

ESPmDNS/library.properties
2:version=3.0.7

BluetoothSerial/library.properties
2:version=3.0.7

NetworkClientSecure/library.properties
2:version=3.0.7

ESP32/library.properties
2:version=3.0.7

Preferences/library.properties
2:version=3.0.7

OpenThread/library.properties
2:version=3.0.7

But for esp8266 version 3.1.2

$ rg version --glob=library.properties
lwIP_w5100/library.properties
2:version=1

SD/library.properties
2:version=2.0.0

lwIP_Ethernet/library.properties
2:version=1

EEPROM/library.properties
2:version=1.0

LittleFS/library.properties
2:version=0.1.0

I2S/library.properties
2:version=1.0

Netdump/library.properties
2:version=2

SDFS/library.properties
2:version=0.1.0

lwIP_enc28j60/library.properties
2:version=1

ESP8266WiFiMesh/library.properties
2:version=2.2

SPISlave/library.properties
2:version=1.0

esp8266/library.properties
2:version=1.0

TFT_Touch_Shield_V2/library.properties
2:version=2.0

Ticker/library.properties
2:version=1.0

ESP8266httpUpdate/library.properties
2:version=1.3

ESP8266AVRISP/library.properties
2:version=1.0

GDBStub/library.properties
2:version=0.3

ESP8266HTTPUpdateServer/library.properties
2:version=1.0

ESP8266WiFi/library.properties
2:version=1.0

ESP8266mDNS/library.properties
2:version=1.2

Servo/library.properties
2:version=1.0.2

lwIP_PPP/library.properties
2:version=1

SPI/library.properties
2:version=1.0

ESP8266WebServer/library.properties
2:version=1.0

ESP8266SSDP/library.properties
2:version=1.0

lwIP_w5500/library.properties
2:version=1

Wire/library.properties
2:version=1.0

DNSServer/library.properties
2:version=1.1.1

ESP8266NetBIOS/library.properties
2:version=1.0

Hash/library.properties
2:version=1.0

ESP8266HTTPClient/library.properties
2:version=1.2

ArduinoOTA/library.properties
2:version=1.0

ESP8266LLMNR/library.properties
2:version=1.0

No such consistency. That's really a problem when the version number for ESP8266HttpClient has been 1.2 since board version 2.5.0-beta1 -- five years worth of changes.

So ignore the 1.2 and make sure to upgrade the board to a later version. Looks like the fix in question is in 2.7.2. But unless you have a reason not to, the latest is 3.1.2 (which is two years old) -- upgrade to that.

10

Thanks.

Dumb question - can you point me to the best way of doing this please? I am using the cloud-based development environment.

I tried to copy the ESP8266HTTPClient .h and .cpp files into my project and change the #include <> to #include "" within my project. This started to compile - but then generated an error with a missing StreamDev.h include file.

This was my suspicion that the latest version relied on dependencies.

What is the simplest way of importing the latest (stable) build into the cloud-based IDE for my board?

Dave

11

All avenues I have explored today lead me to the conclusion that the cloud IDE is supposed to contain the latest stable versions of the board configurations.

I can't see how this is true in my case...

Does that now mean I am stuck and either:

A) Have to download the local copy of the IDE or

B) Switch development to a different board that is supported by the Cloud IDE?

Or (option C) I wonder whether I could edit the offending library to make it 'simply' do what I want it to do?

Dave

12

I tried option (C)...

I edited the ESP8266HTTPClient.cpp file (as part of the library) and modified the HTTPClient::disconnect() method as follows:

if(_reuse /* && _canReuse DER:02092025 */ ) {

I call the setReuse( true ) method as soon as I create an instance of the HTTPClient.

I now get the correct diagnostic/debug message from the execution which appears to leave the tcp port open.

Unfortunately, I still get a 400 error back from my second POST!

Ok, I can now look at my application code again having ruled that out...

Thanks for your help. I have learnt quite a lot about how to track down issues.

If there is any way of updating the ESP8266 board build files (within the cloud tools) I would still be interested to ensure I have the latest source code though.

Dave

13

This topic indicates a similar issue with the esp32 board, stuck on an older version (2.0.17 -- I'm using 3.0.7 with the desktop IDE, and the latest is 3.3.0)

For the 400, there are "echo" endpoints out there that will show the request; as long as you don't mind them seeing those bytes. For example

$ printf "\001\002\003hello\n" | curl -i httpbin.org/anything -H "Content-Type: application/octet-stream" --data-binary @- 
HTTP/1.1 200 OK
Date: Tue, 02 Sep 2025 21:06:50 GMT
Content-Type: application/json
Content-Length: 446
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

{
  "args": {}, 
  "data": "\u0001\u0002\u0003hello\n", 
  "files": {}, 
  "form": {}, 
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "9", 
    "Content-Type": "application/octet-stream", 
    "Host": "httpbin.org", 
    "User-Agent": "curl/8.11.1", 
    "X-Amzn-Trace-Id": "Root=1-68b75c67-1d9aa79c6f1c86d41979cea4"
  }, 
  "json": null, 
  "method": "POST", 
  "url": "http://httpbin.org/anything"
}

BTW, you should probably set an appropriate Content-Type; if nothing else, application/octet-stream for "a bunch of bytes"

14

I have tried a number of different things tonight and all of them have failed to make any difference at all to my second POST response of 400.

I have checked my application through (yet again) and apart from a hashing issue that may (or may not) be occurring within a library for SHA256 this looks fine too.

I think I will give wire shark a go to see what is physically being transferred across the network to see if I can get any further.

If that yields nothing, then it has to be something else in the out-of-date HTTPClient library itself.

I was debating purchasing an ESP32 but, if that build also has issues with old versions of various libraries, then there is not much point.

The reason I am using the cloud IDE is that I can develop the software at home and then deploy it in the Church (where the project is destined for) and make changes to the source code there also. I can also logon using my laptop during down-time at work and make one or two tweaks. Basically, one common online IDE contain the most up-to-date source code.

I did a fairly complex VISCA camera controller for our Church video system using an Arduino Mega2560 - and developed the software in this way. The project is a great success and has been replicated a number of times by other people. My experience with that project is what led me down the ESP8266 route - but, unfortunately, the experience has not been as positive.

I will leave the project for a bit and come back to it again...

Thanks for all the help and suggestions.

And, yes, I did use Content-Type application/octet-stream.

Dave

15

Have you tried

Connection: close
16

There's just not that much going on that can go wrong. HTTP is primarily a line-by-line text protocol, except for when you sending binary payloads. You're sending... 32 bytes, no chunking. There's no auth (authentication nor authorization) going on; the server should take a random request and respond appropriately. Although there's "encryption, decryption and hashing".

Show all the code that sends the payloads.

If you send the same 32 bytes with curl (or equivalent) from your computer, does that work?

As I mentioned, there are software equivalents of Wireshark that will tell you "these are the bytes you sent".

17

SurferTim: Yes, that was the default that started all of this off.

kenb4: I will post the source code etc. over the weekend.

I was looking at a RUST implementation of an application for communicating with the Tapo plugs and noted that they looked at the received header (from the first POST) for the 'Set-Cookie' and then used that as a 'Cookie' header in the second POST (the one that is failing with the 400 return code).

I then went back to the python implementation and noted that the requests.Session() supported persistent cookies under the hood.

Ah, I thought, so I have extracted the 'Set-Cookie' return from the handshake1 POST header and sent it out as a 'Cookie' header with the second handshake2 POST.

I still get a darn 400 error though! I will have a look at what I have done again tomorrow.

The reason I plan to use wireshark (or similar) is that I have a working python script at Church along with a Tapo plug. I plan to capture the transactions for a WORKING python script / Tapo plug pair and contrast that with my ESP8266 / Tapo plug NONE WORKING pair.

At least that way I will have a trace of what 'good' looks like...

Dave

18

Each cookie is set in a separate response header. The attributes to define timeout are either Expires or Max-Age, not TIMEOUT.

You might try the request header

Cookie: TP_SESSIONID=DC5B11FD0FA99FDD7D409FDA947998DA

(Unlike with Set-Cookie, the Cookie request header can have multiple cookies, separated by a semicolon and space)

19

That was exactly my plan for today. We both think alike!

I was looking at the RUST code again last night and it looks like the parsing algorithm is specifically looking for the "TP_SESSIONID" and value pair and ignoring the TIMEOUT.

Dave

20

Well, that was a bit of a letdown - it all worked!

Deleting the ';TIMEOUT=XXX' from the end of the Set-Cookie string and my handshake 2 now returns a POST status of 200...

Cookie: TP_SESSIONID=BEEFA37097C6AA4F3ED2254BAA06A14C

httpResponseCode (handshake2)=200

Right, onwards and downwards...

My next thing is JSON and AES encryption and decryption - and a few more POSTS (but I know all about those tricks now!)...

Thanks for the help. I may be back!

Dave