OTA ESP32 web external

DO NOT WASTE YOUR TIME READING THESE POSTS.

READ POST 27 AND THEN 42

.
.
.
.
.
.
.
.
.
.

I'm back guys

After I watched this video: 34 firmware update Remotely in esp32 (OTA with mDash) - YouTube

I downloaded the library and followed the steps as per the video using this example code:

// See https://mdash.net/docs/ for the full IoT product reference impementation

#define MDASH_APP_NAME "MinimalApp"
#include <mDash.h>

#include <WiFi.h>

#define WIFI_NETWORK "ABC"
#define WIFI_PASSWORD "123"
#define DEVICE_PASSWORD "YSWDv6GHKGDKJGHNBtHg"

void setup() {
  Serial.begin(115200);
  WiFi.begin(WIFI_NETWORK, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("\nWiFi connected, IP address:");
  Serial.println(WiFi.localIP());
  mDashBegin(DEVICE_PASSWORD);
  pinMode(LED_BUILTIN, OUTPUT);
}

void loop() {
  digitalWrite(LED_BUILTIN, HIGH);
  delay(100);
  digitalWrite(LED_BUILTIN, LOW);
  delay(100);
}

But when compiling I got this error: C:\Users\EU\Documents\Arduino\libraries\mDash\src\mDash.c:72:21: fatal error: esp_mac.h: No such file or directory
compilation terminated.

Can't seem to find a dependent library esp_mac.h

NOTE: I use arduino IDE 1.8.16 on Windows 10. ESP32 Wroom.

Does anyone know how to fix?

Thanks

I'm trying to do my first firmware update outside of my wifi network. But for my level of knowledge I find it complicated to do this alone through a server with a subscription. That's why I'm trying the alternatives I see in the searches I do, I also tried with otadrive and it didn't compile either. It gave other types of errors. But it's just a test. Just to see that it really works. Using blink as an example.

If anyone can and wants to help regarding post #1 I would appreciate it. Any other situation like AWS, or MQTT will be more complicated for me.

Can someone post some example of NOTE using some external server?

I'll use blink to test. It doesn't have to be with an SSL certificate.

Amazing that there is nothing on google about this using arduino's IDE

no tutorial

Thanks

I exported the compiled binary and saved it to my google drive.

Using the code below I try to OTA the ESP32 devkit.

Gives error 301 or 302. I added the line: client.setFollowRedirects

it was no use

I'm not worried about https as it's just testing. But I would like to see it work.

Does anyone know how the code should look to work?

Thanks

#include <WiFi.h>
#include <HTTPClient.h>
#include <Update.h>
void updateFirmware(uint8_t *data, size_t len);
// location of firmware file on external web server
// change to your actual .bin location
#define HOST "http://drive.google.com/file/d/1FYNV1XnU0wEsVm9h9bLWP6XBPNzEW"

HTTPClient client;
// Your WiFi credentials
const char* ssid = "xxxxxxxxxxxxxxxxxxx";
const char* password = "zzzzzzzzzzzzzzzzzzzzzzz";
// Global variables
int totalLength;       //total size of firmware
int currentLength = 0; //current size of written firmware

void setup() {
  Serial.begin(115200);
  // Start WiFi connection
  WiFi.mode(WIFI_MODE_STA);        
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  // Connect to external web server
  client.begin(HOST);
  client.setFollowRedirects(HTTPC_STRICT_FOLLOW_REDIRECTS);
  // Get file, just to check if each reachable
  int resp = client.GET();
  Serial.print("Response: ");
  Serial.println(resp);
  // If file is reachable, start downloading
  if(resp == 200){
      // get length of document (is -1 when Server sends no Content-Length header)
      totalLength = client.getSize();
      // transfer to local variable
      int len = totalLength;
      // this is required to start firmware update process
      Update.begin(UPDATE_SIZE_UNKNOWN);
      Serial.printf("FW Size: %u\n",totalLength);
      // create buffer for read
      uint8_t buff[128] = { 0 };
      // get tcp stream
      WiFiClient * stream = client.getStreamPtr();
      // read all data from server
      Serial.println("Updating firmware...");
      while(client.connected() && (len > 0 || len == -1)) {
           // get available data size
           size_t size = stream->available();
           if(size) {
              // read up to 128 byte
              int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
              // pass to function
              updateFirmware(buff, c);
              if(len > 0) {
                 len -= c;
              }
           }
           delay(1);
      }
  }else{
    Serial.println("Cannot download firmware file. Only HTTP response 200: OK is supported. Double check firmware location #defined in HOST.");
  }
  client.end();
  
}

void loop() {}

// Function to update firmware incrementally
// Buffer is declared to be 128 so chunks of 128 bytes
// from firmware is written to device until server closes
void updateFirmware(uint8_t *data, size_t len){
  Update.write(data, len);
  currentLength += len;
  // Print dots while waiting for update to finish
  Serial.print('.');
  // if current length of written firmware is not equal to total firmware size, repeat
  if(currentLength != totalLength) return;
  Update.end(true);
  Serial.printf("\nUpdate Success, Total Size: %u\nRebooting...\n", currentLength);
  // Restart ESP32 to see changes 
  ESP.restart();
}

File access on google drive was set to ME ONLY. Modified to ALL. And after a few minutes it seems that at least one part started to work. And I put https in the HOST.

I was wrong, after showing this data on the serial monitor the ESP32 does not empty if reset, it continues with the same code.

It made the connection showing the 200 but it seems that it did not find the .bin.

I wonder what's wrong ?

whoa.. a 4.2gb fw!! that's some big code, what does it do??
must be a corrupt file or something..
sorry.. ~q

So it is. Too weird.
I copy the HOST URL and paste it in the browser and hit ENTER and it opens the drive.google page with the correct file. I download it and it has 195 Kb.

ip address looks funny too..
if you didn't chop it, where's the rest??

Ah, I cut that one, every time I published a code that I was testing here I forgot to delete the baptismal credentials. But now I haven't forgotten.

You must have drive.google. Don't want to copy this code, paste and test? send a blink.ino there and see what happens.

Leave it there. Don't mind my silly problems.

Suddenly someone comes along who knows how to fix it and wants to help.

Let's wait.

that's what i figured..
it's a class c address so no need to hide it but no worries..

no, no google drive sorry..
going to be tackling ota this weekend, so i'll be watching too..
good luck.. ~q

good weekend

The error of the first code 301 or 302 is that I think it redirects to an HTTPS area and therefore the HTTP library does not have access.

I started testing with this other code that uses WiFiSecury and I tested it on another HOST but now I am facing DNS error problem.

Does anyone know how I can resolve this ?

#define DISABLE_TLS

#include <WiFi.h>
#include <WiFiClientSecure.h>
#include <Update.h>

WiFiClientSecure client;

long contentLength = 0;
bool isValidContentType = false;

const char* SSID = "zzzzzzzzzzzzzzzzz";
const char* PSWD = "xxxxxxxxxxxxxxxxxxxxx";

// HOST
String host = "https://mega.nz/file/FMUBXCRS#lTaGOV95qm4kpO4TkRC1G_Z63PcfzR8WowZ5LNb";
//String host = "https://www.dropbox.com/s/ydse1cmpe5el52/atualiza?dl=0";       //"http://drive.google.com/file/d/1FYNV1XnU0wEsVm9h9bLWP6XNzEWoEZ";
int port = 443;
String bin = "/atualiza.bin";


String getHeaderValue(String header, String headerName) {
  return header.substring(strlen(headerName.c_str()));
}

// OTA Logic 
void execOTA() {
  
  Serial.println("Connecting to: " + String(host));
  
  client.setInsecure();
  
  if (client.connect(host.c_str(), port)) {
    
    Serial.println("Fetching Bin: " + String(bin));

    client.print(String("GET ") + bin + " HTTP/1.1\r\n" +
                 "Host: " + host + "\r\n" +
                 "Cache-Control: no-cache\r\n" +
                 "Connection: close\r\n\r\n");

    unsigned long timeout = millis();
    while (client.available() == 0) {
      if (millis() - timeout > 10000) {
        Serial.println("Client Timeout !");
        client.stop();
        return;
      }
    }
    
    while (client.available()) {
      String line = client.readStringUntil('\n');
      line.trim();

      if (!line.length()) {
        break;
      }

      if (line.startsWith("HTTP/1.1")) {
        if (line.indexOf("200") < 0) {
          Serial.println("Got a non 200 status code from server. Exiting OTA Update.");
          break;
        }
      }

      if (line.startsWith("Content-Length: ")) {
        contentLength = atol((getHeaderValue(line, "Content-Length: ")).c_str());
        Serial.println("Got " + String(contentLength) + " bytes from server");
      }

      // Next, the content type
      if (line.startsWith("Content-Type: ")) {
        String contentType = getHeaderValue(line, "Content-Type: ");
        Serial.println("Got " + contentType + " payload.");
        if (contentType == "application/octet-stream") {
          isValidContentType = true;
        }
      }
    }
  } else {
    Serial.println("Connection to " + String(host) + " failed. Please check your setup");
  }

  // Check what is the contentLength and if content type is `application/octet-stream`
  Serial.println("contentLength : " + String(contentLength) + ", isValidContentType : " + String(isValidContentType));

  // check contentLength and content type
  if (contentLength && isValidContentType) {
    // Check if there is enough to OTA Update
    bool canBegin = Update.begin(contentLength);

    // If yes, begin
    if (canBegin) {
      Serial.println("Begin OTA. This may take 2 - 5 mins to complete. Things might be quite for a while.. Patience!");
      // No activity would appear on the Serial monitor
      // So be patient. This may take 2 - 5mins to complete
      size_t written = Update.writeStream(client);

      if (written == contentLength) {
        Serial.println("Written : " + String(written) + " successfully");
      } else {
        Serial.println("Written only : " + String(written) + "/" + String(contentLength) + ". Retry?" );
        // retry??
        // execOTA();
      }

      if (Update.end()) {
        Serial.println("OTA done!");
        if (Update.isFinished()) {
          Serial.println("Update successfully completed. Rebooting.");
          ESP.restart();
        } else {
          Serial.println("Update not finished? Something went wrong!");
        }
      } else {
        Serial.println("Error Occurred. Error #: " + String(Update.getError()));
      }
    } else {
      // not enough space to begin OTA
      // Understand the partitions and
      // space availability
      Serial.println("Not enough space to begin OTA");
      client.flush();
    }
  } else {
    Serial.println("There was no content in the response");
    client.flush();
  }
}

void setup() {
  
  Serial.begin(115200);
  delay(10);

  Serial.println("Connecting to " + String(SSID));

  // Connect to provided SSID and PSWD
  WiFi.begin(SSID, PSWD);
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print("."); // Keep the serial monitor lit!
    delay(500);
  }
  
  execOTA();
}

void loop() {
}


I commented the #define DISABLE_TLS
and also the client.setInsecure();

and the DNS error no longer occurs but this error occurs here:

Connecting to: https://www.dropbox.com/s/ydse1cmp8e5el52/atualiza?dl=0
[V][ssl_client.cpp:59] start_ssl_client(): Free internal heap before TLS 279516
[E][WiFiClientSecure.cpp:133] connect(): start_ssl_client: -1
[V][ssl_client.cpp:265] stop_ssl_socket(): Cleaning SSL connection.
Connection to https://www.dropbox.com/s/ydse1cmpel52/atualiza?dl=0 failed. Please check your setup
contentLength : 0, isValidContentType : 0
There was no content in the response

Incredible. Don't waste your time typing into google: ESP32 OTA WEB or something like that. ALL examples that are shown, NONE works.

Using the arduino IDE none works. The examples that install Python or use ESP-IDF. MQTT, etc, may work, but using the Arduino IDE... NOTHING.

They don't compile, and when they compile they don't download the .BIN

Of course I will continue the search and if I find it I will post it here. But I do not know when.

I'm trying to do my first OTA off my WiFi network using blink.ino example

Greetings

The closest I've come to success is now with this code.

If you copy the HOST and paste it in your browser you will download the update.bin which is the blink example for ESP32.

I'm not worried about HTTPS, it's just a test

Now I compile successfully, I see a 200 connection. However, the .BIN size that was supposed to be 195 Kb is being recognized as 853. I also don't understand a difference in the date 01 Jan 1970. What I know is that it is crashing right away when it tries do the firmware update. Guru meditation.

#include <WiFi.h>
#include <HTTPClient.h>
#include <Update.h>

// location of firmware file on external web server
// change to your actual .bin location
#define HOST "http://frequencia.infinityfreeapp.com/atualiza.bin"

HTTPClient client;
// Your WiFi credentials
const char* ssid = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz";
const char* password = "zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz";
// Global variables
int totalLength;       //total size of firmware
int currentLength = 0; //current size of written firmware

void setup() {
  Serial.begin(115200);
  // Start WiFi connection
  WiFi.mode(WIFI_MODE_STA);        
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  // Connect to external web server
  client.begin(HOST);
  // Get file, just to check if each reachable
  int resp = client.GET();
  Serial.print("Response: ");
  Serial.println(resp);
  // If file is reachable, start downloading
  if(resp == 200){
      // get length of document (is -1 when Server sends no Content-Length header)
      totalLength = client.getSize();
      // transfer to local variable
      int len = totalLength;
      // this is required to start firmware update process
      Update.begin(UPDATE_SIZE_UNKNOWN);
      Serial.printf("FW Size: %u\n",totalLength);
      // create buffer for read
      uint8_t buff[128] = { 0 };
      // get tcp stream
      WiFiClient * stream = client.getStreamPtr();
      // read all data from server
      Serial.println("Updating firmware...");
      while(client.connected() && (len > 0 || len == -1)) {
           // get available data size
           size_t size = stream->available();
           if(size) {
              // read up to 128 byte
              int c = stream->readBytes(buff, ((size > sizeof(buff)) ? sizeof(buff) : size));
              // pass to function
              updateFirmware(buff, c);
              if(len > 0) {
                 len -= c;
              }
           }
           delay(1);
      }
  }else{
    Serial.println("Cannot download firmware file. Only HTTP response 200: OK is supported. Double check firmware location #defined in HOST.");
  }
  client.end();
  
}

void loop() {}

// Function to update firmware incrementally
// Buffer is declared to be 128 so chunks of 128 bytes
// from firmware is written to device until server closes
void updateFirmware(uint8_t *data, size_t len){
  Update.write(data, len);
  currentLength += len;
  // Print dots while waiting for update to finish
  Serial.print('.');
  // if current length of written firmware is not equal to total firmware size, repeat
  if(currentLength != totalLength) return;
  Update.end(true);
  Serial.printf("\nUpdate Success, Total Size: %u\nRebooting...\n", currentLength);
  // Restart ESP32 to see changes 
  ESP.restart();
}

ERROR

WiFi connected
IP address: 
192.168.18.25
[V][HTTPClient.cpp:245] beginInternal(): url: http://frequencia.infinityfreeapp.com/atualiza.bin
[D][HTTPClient.cpp:293] beginInternal(): protocol: http, host: frequencia.infinityfreeapp.com port: 80 url: /atualiza.bin
[D][HTTPClient.cpp:579] sendRequest(): request type: 'GET' redirCount: 0

[D][HTTPClient.cpp:1125] connect():  connected to frequencia.infinityfreeapp.com:80
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Server: nginx'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Date: Sun, 02 Apr 2023 13:13:24 GMT'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Content-Type: text/html'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Content-Length: 853'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Connection: keep-alive'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Vary: Accept-Encoding'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Expires: Thu, 01 Jan 1970 00:00:01 GMT'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: 'Cache-Control: no-cache'
[V][HTTPClient.cpp:1216] handleHeaderResponse(): RX: ''
[D][HTTPClient.cpp:1257] handleHeaderResponse(): code: 200
[D][HTTPClient.cpp:1260] handleHeaderResponse(): size: 853
[D][HTTPClient.cpp:603] sendRequest(): sendRequest code=200

[D][Updater.cpp:132] begin(): OTA Partition: app1
Updating firmware...
.......Guru Meditation Error: Core  1 panic'ed (LoadProhibited). Exception was unhandled.
Core 1 register dump:
PC      : 0x4000c2e0  PS      : 0x00060030  A0      : 0x80087bfa  A1      : 0x3ffb1da0  
A2      : 0x3ffb1dc0  A3      : 0x00000000  A4      : 0x00000010  A5      : 0x3ffb1dc0  
A6      : 0x00060823  A7      : 0x00000001  A8      : 0x00000000  A9      : 0x00150000  
A10     : 0x3ffbda68  A11     : 0x3ffc1aac  A12     : 0x3ffc1964  A13     : 0x00000000  
A14     : 0x3ffb1e7c  A15     : 0x00000008  SAR     : 0x0000000b  EXCCAUSE: 0x0000001c  
EXCVADDR: 0x00000000  LBEG    : 0x4000c2e0  LEND    : 0x4000c2f6  LCOUNT  : 0x00000000  

ELF file SHA256: 0000000000000000

Backtrace: 0x4000c2e0:0x3ffb1da0 0x40087bf7:0x3ffb1db0 0x4015a919:0x3ffb1e00 0x400d5f86:0x3ffb1e20 0x400d4f1a:0x3ffb1e40 0x400d50e2:0x3ffb1e60 0x400d52ed:0x3ffb1e80 0x400d0c8b:0x3ffb1ec0 0x400d0e43:0x3ffb1ee0 0x400d7592:0x3ffb1fb0 0x40089a6a:0x3ffb1fd0

Rebooting...
ets Jun  8 2016 00:22:57

Tried with ESP DEV MODULE default and DEV KIT V1. Same mistake.

I don't know if you need to format partition. I don't think so because with a USB cable or within my Wi-Fi network it works fine.

If anyone knows how to fix it and wants help...

try modifying this one..

AWS_S3_OTA_Update
does not use https, but slightly different from yours..

looks like your server is sending you text/html instead of a stream..
the above sample first connects then gets, then checks content-type to be proper..
worth a shot, stay off the https, says it doesn't work in the above sample too..

good luck.. ~q