ESP32 HTTPClient.getStream() limits to 1024000 bytes

Hi,

I'm new to this community, though it doesn't feel like it, been visiting this forum for years.
It is a new account though.

I'm having an infuriating problem that I've been struggling with for 3 days.

The idea: Create an OTA method of uploading a new firmware bin, all over secure connections

I already have a WSS client up and running.
So I thought I'd try giving that a go first.
After almost a day of trail and error(cuz there absolutely no support out of the box for that), I gave up on that due to the fact that I cannot find a way to stream the data coming in effectively.
(The program is at 1.3mb compiled and I still have a ways to go, so a LARGE APP).
Theoretically it should be possible by fragmenting the incoming binary data (frames) and then

Update.write()

Presuming MD5 has been set.

I couldn't make that work.

So I attempted an HTTPS GET from the WSS servers' webserver.

This is where I am now...
Status: It uploads and then the stream stops at 1024000 bytes received.
Problem: The stream given by HTTPSClient.getStream(); limits the received binary data to 1mb.

The code is almost 3000 lines long, so I won't attempt to upload it.
I will create an sketch here with the relevant data.

(
If something obvious is wrong, like a varbiable being used before it's declared it's just a copy paste error, the program works
My problem is the stream limit please advise on that and not silly stuff
Note: that the sketch here is just an exerp and hasn't been compiled or tested
)

#include <WiFi.h> 
#include <HTTPClient.h>
#include <WiFiClientSecure.h>
#include <Update.h>
void setup() {

}
void loop() {
if (Update.hasError()) {
    Update.printError(Serial);
    Update.clearError();
  }
  //upgradeOTA is just a global bool to block other events that could interrupt the flashing
  //Also set to true in WSS task to start the flashing
  if (upgradeOTA) {  
    HTTPClient https;
    https.setTimeout(5000);

    // DEBUG
    Serial.print("Target URL: ");
    Serial.println(target_uri);          //SET BY WSS
    Serial.print("FileSize: ");
    Serial.println(upgradeOTA_fileSize); //SET BY WSS
    Serial.print("Given MD5 HASH: ");
    Serial.println(upgradeOTA_MD5);     //SET BY WSS
    Serial.print("");
    Serial.println("");
    // END DEBUG

    https.begin(target_uri,wss_cacert);  //wss_cacert loaded out of eeprom
    Serial.println("  HTTPs Connection Successfull");
    int httpResponseCode = https.GET();
    if (httpResponseCode>0) {
      Serial.print("HTTP Response code: ");
      Serial.println(httpResponseCode);
      Update.onProgress([](unsigned int progress, unsigned int total) {
        //Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
        Serial.printf("Uploaded: %u%%\r", (progress));
        
        if (Update.isFinished()) {    //I assume finished when the specified length is reached
          if (Update.end(true)) { 
            Serial.printf("Update Success: %u\nRebooting...\n");
            ESP.restart();
          } else { }
          Serial.println("Upload Finished");
        }
      });
      Update.setMD5((char*)upgradeOTA_MD5); //upgradeOTA_MD5 SET BY WSS (char[32])
      if (!Update.begin(upgradeOTA_fileSize)) { //upgradeOTA_fileSize SET BY WSS (size_t)
        Update.printError(Serial);
      } else upgradeOTA = false;
  //HERE IS WHERE THE STREAM THAT LIMITS UPLOAD TO 1MB IS RETURNED
      if (Update.writeStream(https.getStream())) { 
        Update.printError(Serial);
      }
    }
    else {
      Serial.print("Error code: ");
      Serial.println(httpResponseCode);
    }
    upgradeOTA = false;
}

I've run through every library's code and I can't find any solution

So my question: Is this a limitation by the client stream like I suspect or might it be somewhere else like on the webserver side?
And: How, if possible, do I set the stream limit for a HTTP GET request.

If you think I should go back to trying to do this in WSS please provide a working snippet of a firmware flash that is LARGE(like at least 1.5mb capable)

PS: It uploads perfectly and gives Serial feedback all the way up to 1024000bytes received then stops and after the timeout expires it gives a serial error saying

"Stream Read Timeout"

Thank You for reading my 'assay' and for the help provided.

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.