Arduino OTA breaking my code

Hello, I'm currently trying to use Arduino OTA and I can't understand some behavior

Just for you to understand my project:
It is an Arduino Telegram Bot which can receive and send message. I added a feature that allow me to update the Arduino by sending a file on Telegram. The functions to send, receive et download the message work perfectly! Until... until I use OTA.

Let me explain:

  1. If I don't use the part of the code that allow me to update the Arduino, every goes well.
  2. If I call that function to update the code by OTA, then the Arduino starts behaving oddly (I'll explain)

Here is my code (only the part with the OTA update):

InternalStorage.debugPrint();

if (InternalStorage.maxSize() > File_size)
{
  if (InternalStorage.open(File_size))
  {
    InternalStorage.debugPrint();

    int b;

    while (File_size > 0)
    {
      b = Fichier.read();

      if (b != (-1))
      {
        InternalStorage.write(b);
      }
      else
      {
        InternalStorage.close();
        break;
      }

      File_size--;
    }

    //InternalStorage.close();

    if (File_size == 0)
    {
      //Envoi_Message_Telegram(Nouveau_Message_Telegram.ID_Conversation, "L'arduino va redémarrer avec le nouveau programme");
      //delay(1000);
      InternalStorage.close();
      InternalStorage.debugPrint();
      //InternalStorage.apply();
      Serial.println("Test1");

      return Retour_FCT;
    }
  }
  else
  {
    Serial.println("Echec ouverture mémoire");
  }
}

Don't worry for the fact that there is no return in certain conditions; there is one return after all this part.

So I want this code to work this way:

  1. Receiving the file (and so its size) ; it works!
  2. Looking if there is enough place. If not, then it's the end. If yes:
  3. Opening the memory and then writing the bytes in the good place to work
  4. If there was an error during reading the file, we break the procedure
  5. If not, then we should apply the changes.

Applying the changes seems to work so it's good. But before applying the changes I wanted to send a message on Telegram to say that it's going to reboot; but it doesn't work. I then tried to print some messages over Serial.println() and something seems broken.
The first InternalStorage.debugPrint(), which is BEFORE opening the storage works and the output is

19:07:47.287 -> SKETCH_START_ADDRESS 8192
19:07:47.287 -> PAGE_SIZE 64
19:07:47.287 -> MAX_FLASH 245760
19:07:47.287 -> MAX_PARTIONED_SKETCH_SIZE 118784
19:07:47.287 -> STORAGE_START_ADDRESS 126976

But the second one after opening the InternalStorage isn't as good:

19:07:51.113 -> SKETCH_START_ADDRESS 8192��PAGE_SIZE 64��MAX_FLASH 245760��MAX_PARTIONED_SKETCH_SIZE 118784��STORAGE_START_ADDRESS 126976��

The LF and CR don't work anymore... And it is the same with all my Serial.println() that come after it, even in other parts of the global sketch.
Moreover, after that the Arduino seems to continue to execute the code but it doesn't work, if I send a message on Telegram then nothing happens

Finally, even cliking on the reset button doesn't work. The arduino disconnects from the computer but never connect back. I have to click twice quickly on the reset button to be able to upload another sketch.

Has someone an idea on how to fix that? Is someting missing in my code?

what is file and when its read returns -1?

Just for you to know

If I use the SAME functions without touching a thing (except mistake from me), but deleting all other functions (for example I use this Arduino to command my garage door); then everything works as expected and I can send Telegram message before applying the changes.

Do you think that the fact the sketch is smaller impacts it? Even if I verify if there is enough place before writing the file...

File (or 'Fichier' in my code since I'm french) is a WiFiSSLClient
I read directly from that to copy on the Arduino memory
Honestly I don't know in which case it returns (-1), but I'm sure it doesn't because at the end File_Size equals 0.
And even if we admit that the file isn't valid, it doesn't matter here right? Since I don't apply the changes.

BTW, the binary file I send is the blink example

what is the size of the running sketch?

The sketch that works is 70608 bytes (26% of the memory)
The one that doesn't work is 121008 bytes (46%)

And the bin file I send (Blink.ino but compiled) is 15132 bytes

Why the question?

so I don't know why the running sketch is corrupted, but I guess it is because of the size even if the address math says it is safe.
but you will not be able to OTA upload that sketch.
MAX_PARTIONED_SKETCH_SIZE 118784

Ok but the lib says me that there is enough place... So even if at the end there is not enough place (because of I don't know what)
Why right after opening the memory, before I even right a signle byte, the serial starts bugging?

Another question pls,
When we upload from the computer, then the sketch can take 90% of the memory if we want to
When we use OTA, is there a limit? Even with Arduino cloud?

So the addresses math is not OK.

ArduinoOTA uses upper half of the flash for storage so the sketch size is limited. And there are 8192 bytes of bootloader at the beginning of the flash memory.
So 8192+121008=129,200, but the storage starts at 126976 so the the end of the running sketch is erased.

Cloud OTA stores the OTA binary in the NINA. Then the Cloud sketch with OTA has the SNU library which prepends a second stage bootloader, which reads the binary from NINA storage.

What does it mean that the addresses math is not OK? Can I do something about it? Or is it from the library that is malfunctionning?
Maybe I can add an offset before writing so that it doesn't modify that actual sketch?

Sorry but I don't get the difference with Cloud OTA? Does it mean that the limit of the size of the sketch is the same as if we were plugged on the computer?

If we look at the the numbers, we can see that STORAGE_START_ADDRESS is equal to SKETCH_START_ADDRESS + MAX_PARTIONED_SKETCH_SIZE.
And after some test, MAX_PARTIONED_SKETCH_SIZE doesn't change, it doesn't depend on the sketch I uploaded in the Arduino.
So maybe that the value hasn't been well defined? Or it should be a constant?

I looked into the library and I saw where those values are defined:

InternalStorageClass::InternalStorageClass() :
  MAX_PARTIONED_SKETCH_SIZE((MAX_FLASH - SKETCH_START_ADDRESS) / 2),
  STORAGE_START_ADDRESS(SKETCH_START_ADDRESS + MAX_PARTIONED_SKETCH_SIZE)

If we look at my previous post, according to the library, MAX_FLASH is 245,760 bytes; but when I compile my sketch on Arduino IDE 2.X, here is what I got:

The sketch uses 121,200 bytes (46%) of program storage space. The maximum is 262,144 bytes.

Global variables use 5,716 bytes (17%) of dynamic memory, leaving 27,052 bytes for local variables. The maximum is 32,768 bytes

We can see that the maximum size isn't the same... 262,144 bytes according to the IDE; could this be the problem?

I got some news:

After a lot of tests, I understood why there were some bugs during the process. Indeed, the WiFi module wasn't fast enough to download the bytes compared to the speed of writing in the memory of the Arduino. I then added those lines during the writing process:

unsigned long Attente_Packets_Fichier = millis();
while ((Fichier.available() == 0) && (millis() - Attente_Packets_Fichier < 2000));
Attente_Packets_Fichier = millis();

This way, if we red all the bytes available, but not all the file, we wait.

However, this doesn't explain why my code was behaving oddly when the update went not well because if I don't apply anything, then nothing should change. I get what @Juraj said (about erasing the end of the current program) but the thing is that I tried with a smaller "based" code
with only the OTA and Telegram functions. It is 39,808 bytes long so we are way far from the STORAGE_START_ADDRESS but it kept bugging.
BTW, I tried to modify its value by adding 8192 so that the STORAGE_START_ADDRESS is equal to 135,168 but it didn't change a thing (adding 8192 is a choice: if we considere the IDE is right, then MAX_FLASH is 262,144 bytes instead of 245,760)

Also, I think there are some problems in the open() function because sometimes the Arduino blocked in there, not going any further in my code (so IDK where it's stuck, but it is somewhere during the open() function since the Serial.print() I put right after don't doesn't print.

According to me there is also a problem in the close() function because if I don't apply the changes, then nothing should be modified and this is clearly not the actual thing...

And finally, it sound weird to me that the values such as MAX_FLASH are different between the library and the IDE.

Beyond all of this, the lib works very good and the update goes well each time I apply the changes (if I collected all bytes of my file of course). But do you think you can do something about it? (You or someone else)

the eeprom emulation space takes the rest of the flash

MAX_FLASH(PAGE_SIZE * NVMCTRL->PARAM.bit.NVMP - eepromSizes[EEPROM_EMULATION_RESERVATION])

where the eeprom size is read from fuses

#  define EEPROM_EMULATION_RESERVATION ((*(uint32_t*)(NVMCTRL_FUSES_EEPROM_SIZE_ADDR) & NVMCTRL_FUSES_EEPROM_SIZE_Msk) >> NVMCTRL_FUSES_EEPROM_SIZE_Pos)

So the EEPROM Emulation is at the end of the program? Since the bootloader is 8192 bytes and my sketch starts at 8192? Maybe that why it breaks the Arduino... But it doesn't explain why it starts writing the bytes OVER the actual sketch. Do you think something can be done about that?

Just, I quote this because I'd like to know pls:

yes
they use WiFiStorage.downloadOTA(ota_url, &nina_ota_err_code)

1 Like

That is so bad that we can't co it by the way I use it!
This way I would have no limit and maybe no bug

what bug?

the fact that from the moment I opened the InternalStorage then I can't avoid to apply the change, otherwise my Arduino won't be working normally
But a fail during the download of the file can happen so... it will break my code