Https Mystery: how to download like a browser

I would like to download bitmaps through https without the need for preloaded certs.

The goal is to use a general solution for GxEPD2_WiFi_Example for ESP8266 and ESP32.

Browsers like Microsoft Edge or Mozilla Firefox can do this. With Firefox I found out how to get the certificate chain. But certificates have expiration dates, and thus need to be updated.

Using setInsecure() on WiFiClientSecure no longer works for all cases.

So far, I found these examples to learn from: WiFiClientShowPeerCredentials and BearSSL_Validation for ESP32 andESP8266.

For WiFiClientShowPeerCredentials I had to add a preloaded certificate to get it working, and it got me no further (with some additions).

With BearSSL_Validation I found out how you can use a root certificate, but there doesn't seem to be a common root for all sites.

I intend to add more to this post, but any hints would be highly welcome.

Jean-Marc

Browsers and operating system update their certificate list…

If you use https you need crypto capabilities and the right certificate

But how do they do this?

Don't ESP32 and ESP8266 have these capabilities? How do I get the right certificate without preload?

WiFiClientShowPeerCredentials was obviously intended to show how to do this, but it no longer achieves that goal, as it no longer works with setInsecure().

Add the ability to get the peer certificate of an SSL connection; use…
…ful for IoT when the root/cert trust chain has a shorter lifecylce than the device itself.

You get them through OS software updates through signed sources (download from Apple or M$FT)

Thank you. Yes, I understand this. They have (updated) root certificate(s) I guess.
Looks like I need to investigate further, to understand how this all goes together.

It looks like BearSSL of WiFiClientSecure of ESP8266 can do more than WiFiClientSecure of ESP32.

Still more hints are welcome.

Jean-Marc

Maybe I just discovered where I got stuck in the example WiFiClientShowPeerCredentials.
The relevant line of code I looked at is:

  const mbedtls_x509_crt* peer = client->getPeerCertificate();

But this mbedtls_x509_crt only contains part of the information.
There is more to investigate:

class WiFiClientSecure : public WiFiClient
{
protected:
    sslclient_context *sslclient;
 
    int _lastError = 0;
    int _peek = -1;
    int _timeout;
    bool _use_insecure;
    const char *_CA_cert;
    const char *_cert;
    const char *_private_key;
    const char *_pskIdent; // identity for PSK cipher suites
    const char *_psKey; // key in hex for PSK cipher suites
    const char **_alpn_protos;
    bool _use_ca_bundle;

some reading

Core documentation

BearSSL_CertStore example;

1 Like

Thank you very much. I don't know why I didn't look at that example yet.

// However, if you don't know what specific sites the system
// will be required to connect to and verify, a
// CertStore can allow you to select from among
// 10s or 100s of CAs against which you can check the
// target's X.509, without taking any more RAM than a single
// certificate.  This is the same way that standard browsers
// and operating systems verify SSL connections.

I think this already answers most of my questions. And I will take a look at the core documentation.

Jean-Marc

great. have fun (may be share your findings, I'm sure it will be helpful)

Next hurdle to pass is in certs-from-mozilla.py

# check if ar and openssl are available
if which('ar') is None and not os.path.isfile('./ar') and not os.path.isfile('./ar.exe'):
    raise Exception("You need the program 'ar' from xtensa-lx106-elf found here: (esp8266-arduino-core)/hardware/esp8266com/esp8266/tools/xtensa-lx106-elf/xtensa-lx106-elf/bin/ar")
if which('openssl') is None and not os.path.isfile('./openssl') and not os.path.isfile('./openssl.exe'):
    raise Exception("You need to have openssl in PATH, installable from https://www.openssl.org/")

raises exception for ar.
I don't know where to find it. The path may be outdated or for Unix.
I also had problems with some certUpdate and certUpdate.py not working on Windows 10.

1 Like

On my Mac it's here

/Users/XXX/Library/`Arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/3.0.4-gcc10.3-1757bed/xtensa-lx106-elf/bin/ar

On your PC you could check

c:\Users\XXX\AppData\Local\Arduino15\packages\esp8266\tools\xtensa-lx106-elf-gcc\3.0.4-gcc10.3-1757bed\xtensa-lx106-elf\bin\ar.exe

Yes, it is there! Thank you. I was too blind to see. I looked too far down the directory tree, maybe.

But this is now more an exercise to learn from, as it is too complicated for a simple example.
And my example needs to work on both ESP32 and ESP8266.

At least I learnt why all simple examples use matched certificates to preload and none has a general solution, because browsers need to handle a large amount of CA certificates.

Exactly !

1 Like

therefore, I tried to get one step ahead. But I caught the next exception:

if which('openssl') is None and not os.path.isfile('./openssl') and not os.path.isfile('./openssl.exe'):
    raise Exception("You need to have openssl in PATH, installable from https://www.openssl.org/")

I don't know what to install, as there is no installable package there for Windows 10.

I expected that ssl is part of python3, so I commented out these two lines.
But I don't know what to change this line to

    ssl = Popen(['openssl','x509','-inform','PEM','-outform','DER','-out', certName], shell = False, stdin = PIPE)

and it raises an exception in a module I didn't expect:

C:\ZinggJ\Arduino\Demos_Tests\ESP8266\BearSSL_CertStore>python3 certs-from-mozilla.py
AC Camerfirma, S.A.:AC Camerfirma SA CIF A82743287:http://www.chambersign.org -> data/ca_000.der
Traceback (most recent call last):
  File "certs-from-mozilla.py", line 65, in <module>
    ssl = Popen(['openssl','x509','-inform','PEM','-outform','DER','-out', certName], shell = False, stdin = PIPE)
  File "D:\obj\Windows-Release\37win32_Release\msi_python\zip_win32\subprocess.py", line 775, in __init__
  File "D:\obj\Windows-Release\37win32_Release\msi_python\zip_win32\subprocess.py", line 1178, in _execute_child
FileNotFoundError: [WinError 2] The system cannot find the file specified

Any help would be welcome, for me to go on learning.

Jean-Marc

I can read python scripts, but obviously don't understand the content.
Popen seems to try to open a command line program openssl, so I need to find one such.

There is a list here: Binaries - OpenSSLWiki, but which one to install? Are they safe?

On Linux

apt-get install libssl-dev libffi-dev

On MacOS

brew install openssl

And if you need to install homebrew

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Thanks for the help of @J-M-L I got a CertStore working on ESP8266.

I used this site to download and install openssl: https://kb.firedaemon.com/support/solutions/articles/4000121705
I downloaded the windows installer version:
Windows installer which will deploy OpenSSL 3.0 x64 (64-bit)

I created a certificates archive certs.ar as provided by BearSSL_CertStore example.
It is available in GxEPD2_WiFi_CertStore_Loader. And can be loaded to ESP8266 LittleFS (SPIFFS) with this utility example.

I rudimentarily tested the certs.ar on ESP8266 using BearSSL_CertStore.

Next step will be an example GxEPD2_WiFi_CertStore_Example for showing BMP files on e-paper displays, a variant of GxEPD2_WiFi_Example using the CertStore.

Jean-Marc

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.