Esp8266 - https connection with client.setInsecure()

Hello ESP Experts,

I have problems establishing an "unsecure" https connection.

In my sketch I call the Spotify API with the following code:

  const char* fingerprint = "XXXXXX"; 

  HTTPClient https;
  
   https.begin("https://accounts.spotify.com/api/token", fingerprint);
  
   https.addHeader("Content-Type", "application/x-www-form-urlencoded");
    
   String completeAuth = "grant_type=authorization_code&code=XXXXX&redirect_uri=XXXXX&client_id=" XXXXX&client_secret=XXXXX";
     
    int httpCode =  https.POST(completeAuth);
    
    Serial.println(httpCode);

This works fine!

Running this code in a program with other tasks running simultaniously I get heap problems and the ESP crashes. So I wanted to try a unsecure connection (I know the risks) but I don't get it running.
I found some examples and gave it try with this code:

   const char* fingerprint = "XXXXXX"; 

   HTTPClient https;

   BearSSL::WiFiClientSecure testclient;
   testclient.setInsecure();
  
   https.begin(testclient, "https://accounts.spotify.com/api/token");
  
   https.addHeader("Content-Type", "application/x-www-form-urlencoded");
    
   String completeAuth = "grant_type=authorization_code&code=XXXXX&redirect_uri=XXXXX&client_id=" XXXXX&client_secret=XXXXX";
     
    int httpCode =  https.POST(completeAuth);
    
    Serial.println(httpCode);

This doesn't work, I always get a "-1" as result. To get behind this, I also tried establishing a regular https connection with testclient.setFingerprint(fingerprint); instead of testclient.setInsecure(); , but this doesn't work, too. I don't know why.

I also tried a another https.begin syntax as described here - without any success.

Any help or explanation would be appreciated - thanks in advance!

I always struggle with https (never really needed it myself so far) but

Running this code in a program with other tasks running simultaniously I get heap problems and the ESP crashes

What other tasks ? there should be no reason for other tasks to interfere and setting the connection insecure will probably not fix it either. Anyway, can you elaborate ?

Deva_Rishi:
...but What other tasks ? ...

Thanks for your answer!
I have a webserver open and a led matrix running. There is no doubt it's a lack of heap.

I dont't struggle with https itself. I just want to know how I can get a working unsecure connection with client.setInsecure() - this is a question independent from my original sketch.

I have a webserver open and a led matrix running. There is no doubt it's a lack of heap.

Yes there is ! On an ESP there is so much available memory that heap issues rarely occur, and if you led-matrix consists of ws2811 or other individually addressable chips, the cause is very likely something else. Changing anything to the webserver or it's connection will not fix your issue in that case.
So how are you driving the led-matrix ?

Deva_Rishi:
Yes there is ! On an ESP there is so much available memory that heap issues rarely occur[...]

That really sounds good! I'm far from being an expert and heap was the only explanation for my issue I could find.

My display is driven by the great Parola library. My program is very large and so I cannot post the complete code here in the forum; but it can be found at my github repo.
The program was written for ESP8266 and ESP32 and it works fine on both devices - except the https connection which works on ESP32 only. That is the reason why I came to the 'lack of heap' hypothesis.

I understand your hypotheses, but you don't extensively use the heap really, you use the stack mainly. How much of your memory is being used by global variables ?
Agreed you program is 'huge' but i am rather confused about what could be the point of this :String speedi = server.arg(five).c_str();why do you need to convert it to a c-string if you store the result in a 'String' thereafter ? As before, you mainly use c-strings for storage, but really quite extensively, all together that must accumulate to quite a lot. Particularly this :

const uint8_t tablesize = numofarticles*newssources;               // (number of articles * number of newssources) defines size of maintable
                                                      struct maintable                        // table for weather and news
{
 char wetcur[500];                      // current weather strings
 char newscur[500];                     // news strings
 char newslink[200];                    // news url strings
};

maintable table[tablesize];             // number of elements specified by (newssources * numofarticles) or (numberofcities*2)  - d

is over a kb per entry and due to the nesting within the declaration it is not very transparant how much is actually declared, still i thing it should all fit in quite easily though.
what is actually the result of this ?const uint8_t newssources = (sizeof (ownNewsSources)/sizeof (ownNewsSources[0])); And finally, what is the size of you display, i couldn't easily find the memory allocation for it in the MAX72xx files, but the buffer was probably put on the heap at 3 bytes per pixel. You may be right it might be a heap issue, it seems that the library was written for chips that use a clock pin as well, which don't need disabling of the interrupts, but that doesn't mean that your display is actually of that type, or that the interrupts may not get turned 'off' anyway. It is quite a lot of code to sift through. Do you really expect a lot less heap usage from not using the secure connection ?

Thanks a lot for taking a look at my code.

I know that I use large c-strings, but I need them to save news and weather data called from server - they can have this size. I can't avoid these large strings beacuse this would restrict the main purpose of my program.

How much of your memory is being used by global variables ?

Arduino IDE says 69%.

String speedi = server.arg(five).c_str();

That's indeed unnecessary. I started writing my program some months ago as a real beginner and it's possible there are some oddities in my code left. But I doubt this affects the https issue because this function is "manually" called and so, it shouldn't interfere with the https connection.

const uint8_t newssources = (sizeof (ownNewsSources)/sizeof (ownNewsSources[0]))

Writes the number of elements in array 'OwnNewsSources' into the variable 'newssources'. I don't know if this is very elegant, but it works.

And finally, what is the size of you display, i couldn't easily find the memory allocation for it in the MAX72xx files, but the buffer was probably put on the heap at 3 bytes per pixel

I use eight MAX7219 8x8 LED modules.

Do you really expect a lot less heap usage from not using the secure connection ?

That's my guess, because my functions 'getWeatherData' and 'getNewsData' establish 'normal' http connections following the same pattern (Server call with http client, result as json). The only difference: The Spotify https call uses a http.POST; could this be reason, that neither client.setInsecure() nor client.setFingerprint() do work?

Thanks a lot for your helpfulness!

I use eight MAX7219 8x8 LED modules.

I only just now saw that these modules are monochrome, but even if they weren't it could hardly be the issue, but at 8x8x8 bits the buffers is very small (even X24 ith would be insignificant)

But I doubt this affects the https issue because this function is "manually" called and so, it shouldn't interfere with the https connection.

Agreed most definitely not the issue, it just sprung out at me.

Writes the number of elements in array 'OwnNewsSources' into the variable 'newssources'. I don't know if this is very elegant, but it works.

that is an array of String * i sort of got that, the thing is that you use the value later to create the array of structs. Just now i am starting to think that you may be writing beyond the size of the array at some point, though that doesn't have anything to do with the https server at all i guess. Also that hardly explains the ESP32 not having the issue at all. Which brings to mind, what ESP core are you using ? have you tried any of the older ones ? I've been having strange issues with 2.6.x and in fact just stuck with 2.4.2 (may have been my older python version causing the issue, but still) There is a few bugs in there still !

Arduino IDE says 69%.

That means 31% of 4MB is left, that is more than 1 MB, which is way more than you should need.

The Spotify https call uses a http.POST; could this be reason, that neither client.setInsecure() nor client.setFingerprint() do work?

maybe... as i said i don't really have much experience with https (or even posting for that matter) i have used it for testing purposes but that is all.
I do have a recollection of not being able to get that to work either, and once the correct fingerprint was used in the constructor (or begin() ) it didn't matter anymore cause we (i was helping someone) got it to work.
About the ESP-core i have run into 1 issue while transporting code from an AVR, that division by zero actually crashes the ESP, when it doesn't crash an AVR (it should really, so which core has the 'bug' is a matter of opinion) and there are possibly a few more anomalies between the ESP32 and the nodeMCU.
I still don't think the heap is the issue, but it probably is related to memory. Most of the ESPserver functions use String extensively, also since the core does quite a good job of managing the 'String' class memory allocation. A String has an undefined size at compile time and even at runtime, and storing the results from these functions in a char-array is running the risk of writing beyond the size of the array, so you have to be really careful with that.
When i looked through your code again it appears that the repo version is not the version that crashes (not the one with https, cause that should include ESP8266ClientSecure no ?)
Anyway these were my thoughts, drop the setInsecure() the https should work as normal, but try a different core version, just to check.

Just now i am starting to think that you may be writing beyond the size of the array at some point[...]

I think I can exclude this possibility. In this respect the code is much more than double checked and works fine in every constellation - running for several days without crashes, restarts etc.

When i looked through your code again it appears that the repo version is not the version that crashes (not the one with https, cause that should include ESP8266ClientSecure no ?)

Unfortunately, the code posted at the repo shows this behaviour - on an ESP8266 the https connection makes the device crashing. That was the reason, why I wanted to try with an insecure connnection. So, at the moment, there is no working way to get a https connection running combined with the other parts and functions of my program.

That's not a catastrophe for me, because it works fine with an ESP32. I just was interested if there is an way to get this running on ESP8266 and so I had the idea to test it with an unsecure connection...

Thanks a lot for your answers and best regards!

Unfortunately, the code posted at the repo shows this behaviour - on an ESP8266 the https connection makes the device crashing.

using this method ? I must have missed it in the code somewhere.