Install https certificate for Arduino ESP32 SERVER

Hi all,
I need the community help for a widely discussed topic, but the answer to my question is not easy to find...
I developed a very simple HTTP server thet lives in my house, under my ADSL router (with proper port forwarding...) and reachable with a dynamic DNS service (needed for the dynamic public IP that my ADSL provider only supports).

The purpose was to reach my server from anywhere, using a generic web browser, just typing "myServerDomain.duckDNS.org:PORT/readValue"... All good, well done man, great job!!...
BUT...
now I need to switch to an httpS server and the game is very different.
I tried to buy a certificate (from cheapSSLweb C.A.) but I got stuck in the validation procedure. (they need a proper domain, with associated email and bla bla bla...) So I had to cancel it.
Now, my question is: anyone of you have eved done something similar?
Where can i buy a certificate in the arduino-compatible format?
How can I "install" it?

EDIT: I'm not posting any code just because my problem is not in "coding" but in "knowing what to code" let's say...

Two main issues:

  • where will TLS/SSL actually run
  • getting the cert(s)

Seems like you can enable TLS directly with the web server running on the ESP32. Here's a fairly recent thread trying to enable it on ESPAsyncWebServer, which says it doesn't quite work. One message points to another server, HTTPS_Server_Generic, which has since been archived (no more updates) almost two years ago.

You'll need the key pair: certificate and private key, and maybe a password for the key. There would be an API to load those bytes, and if it works, it works.

The alternative is a TLS termination proxy that handles that part, and then forwards the unencrypted traffic to your ESP32, which doesn't need any changes. You'd want to reconfigure your network so that the unprotected ESP32 is only reachable from other devices in your network. If you have another computer handy to run the proxy, that would be easier.

Once you have somewhere to run the TLS, then yes, you'll need a proper domain. The certs are valid only for given hostnames, which may include wildcards; e.g.

  • example.com
  • *.example.com
  • www.example.com

A cert for *. would cover www. -- whether that is appropriate is a "that depends" question. I don't know if you'll be able to get a cert for specifically myServerDomain.duckDNS.org; and whether duckDNS.org has anything to say about it. If not, they you'll need to get your own domain, e.g. myserver.example; then for that domain configure its CNAME to be myServerDomain.duckDNS.org

Using a termination proxy has the likely added advantage of being easier to configure and update the certs, which may have to be updated like every 90 days.

Kenb4, thank you for your explanation!
I don't have the option to use a second device as a termination proxy, so I have to proceed with the other option.
I had a look to the library you mentioned and I'll try to use it. Do you have any experience in where to buy a certificate that does the job? (I don't need any extra features; I just need to enable HTTPS requests...) My fear is to buy a certificate in a format that is not what I really need...
Thank you again.

Regarding the library HTTPS_Server_Generic, if I understand well, in the examples I see:

// Include certificate data (see note above)
#include "cert.h"
#include "private_key.h"

// We will use wifi
#include <WiFi.h>

#include <HTTPS_Server_Generic.h>

// The HTTPS Server comes in a separate namespace. For easier use, include it here.
using namespace httpsserver;

// Create an SSL certificate object from the files included above
SSLCert cert = SSLCert(
                 example_crt_DER, example_crt_DER_len,
                 example_key_DER, example_key_DER_len
               );

// Create an SSL-enabled server that uses the certificate
// The constructor takes some more parameters, but we go for default values here.
HTTPSServer secureServer = HTTPSServer(&cert);

where I find "cert.h" and "private_key.h".

Theese files contains:

#ifndef PRIVATE_KEY_H_
#define PRIVATE_KEY_H_
#error You have to run the script extras/create_cert.sh to recreate these files
#endif

and

#ifndef CERT_H_
#define CERT_H_
#error You have to run the script extras/create_cert.sh to recreate these files
#endif

so they say that I have to run the "create_cert.sh" script to fill them with my certificate parameters... (I guess.....)
Up to this point is almost clear, but what next?

Where do I have to put the stuff that I receive when I buy the certificate?

Thank you again and again....

It's easier to see if you run that create_cert.sh script, but there's a simple bug in it. Find it first. In the shell, go to the libraries folder in your sketchbook. The default location of the sketchbook depends on your OS, but it's something like ~/Documents/Arduino. The script is also a Bash script and uses utilities like openssl and xxd, so if you're on Windows, you need to figure that out. (Running WSL is one solution; you can link your mounted C: drive into there.)

$ cd ~/Documents/Arduino/
$ cd libraries/HTTPS_Server_Generic/extras/
$ vi create_cert.sh

(Or whatever editor you use.) Lines 69-70

# Copy files to every example
for D in ../examples/*; do

At some point all the example .ino files got organized into subdirectories and moved down one level, so it should be

for D in ../examples/*/*; do

Save and exit. The downloaded library files are not executable by default, so enable it and run it

$ chmod +x create_cert.sh
$ ./create_cert.sh
Generating RSA private key, 1024 bit long modulus
...
Adding certificate to example Static-Page
Adding certificate to example Websocket-Chat

Certificates created!
---------------------

  Private key:      private_key.h
  Certificate data: cert.h

Now open an example, like HTTPS_Server_Generic > ESP32_WiFi > Static-Page, and examine the cert.h file. It should now start with something like

#ifndef CERT_H_
#define CERT_H_
unsigned char example_crt_DER[] = {
  0x30, 0x82, 0x02, 0x18, 0x30, 0x82, 0x01, 0x81, 0x02, 0x01, 0x02, 0x30,
  0x0d, 0x06, 0x09, 0x2a, 0x86, 0x48, 0x86, 0xf7, 0x0d, 0x01, 0x01, 0x0b,

If you examine the script file, it

  1. creates a fake CA (Certificate Authority)
  2. creates a new key pair
  3. signs the pair with the CA
  4. converts the resulting PEM (text) files to binary DER format
  5. generates the .h files, using xxd to convert the bytes into a C statement
  6. runs that (fixed) loop to copy the results into the examples

One last wrinkle: the library requires OpenSSL. If you don't have it, it will fail when compiling because in a few places, it has

#include "openssl/ssl.h"

The current arduino-esp32 v3 board platform does not have it. But for example, my vendor-specific M5Stack platform running off a fork of v2 does. (The board platform files are off the OS-specific Arduino15 directory; this is a Mac)

$ cd ~/Library/Arduino15/packages/
$ find . -name ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32c3/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32c6/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32h2/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32s3/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./esp32/tools/esp32-arduino-libs/idf-release_v5.1-33fbade6/esp32s2/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32c3/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32c3/include/openssl/include/openssl/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32/include/openssl/include/openssl/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32s3/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32s3/include/openssl/include/openssl/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32s2/include/mbedtls/mbedtls/include/mbedtls/ssl.h
./m5stack/hardware/esp32/2.1.2/tools/sdk/esp32s2/include/openssl/include/openssl/ssl.h

If you scroll to the right, esp32 has mbedtls, but not openssl; m5stack has both. Can you substitute one for the other? I have no idea.

If you add the usual WiFi credentials to the Static-Page example and Upload it, it should print something like

HTTPS WiFiWebServer is @ IP : 192.168.1.15
To access, use https://192.168.1.15
Starting server...
Server ready.

As mentioned in a few places, the self-signed cert cannot be validated, but it otherwise fulfills all the requirements. So if you use a browser, you'll get a warning. And if you use curl, you'll need to use the --insecure switch, which is also -k (-i adds the status-line and headers)

$ curl -i https://192.168.1.15
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
$ curl -ik https://192.168.1.15
HTTP/1.1 200 OK
Content-Type: text/html

<!DOCTYPE html>
<html>
<head><title>Hello World!</title></head>
<body>
<h1>Hello World!</h1>
<p>Your server is running for 35 seconds.</p>
</body>
</html>

So why do you need HTTPS at all? If you want to ensure that the server is not being spoofed, then you when you get the key pair files, they'll be either in PEM or DER format. You can use the same commands in create_cert.sh to convert those into the .h files.

Let's Encrypt is free; there are others.

But if you just need the connection to be encrypted, then the clients usually have a way of saying: don't bother validating. ESP32 NetworkClientSecure/WiFiClientSecure has setInsecure, like curl; you just turn it off. Knowing the risks, that may be acceptable for personal use.

Wow, I have to study your answer in order to undertand it! :slight_smile:
Anyway, you asked "why do I need HTTPS", here the answer: I need it becaure I need to share via Whatsapp a link like "http://myPublicIP:PORT/command", but if I share a link like this, the browser, while opening the link, adds "s" for safety reasons. So the link become "httpS://BlaBlaBla..." and if I don't have a trusted https server running on my arduino, the request doesn't reach it. (The link is used by "non-tech" people, so the option to accept the browser worning and "continue anyway", is not acceptable...)
All this mess for a very little thing...