OTA upload freezes

I have three Nano 33 IoT systems and two of them are exhibiting this problem. Neither system can be updated from Web Editor using USB or OTA.

One is a test system so I can mess with it. I can upload a blink sketch from Web Editor, but as soon as I include the IoT Cloud libraries, even if the Thing is trivial, it produces the linker warnings and the upload fails.

The failure mode is different between USB and OTA. With USB it does not seem to grab the attention of the Nano and (Verifying...) reverts to the verify / upload buttons as soon as the compile is complete.

After using IDE 2.3.2 to upload a trivial cloud sketch, I can attempt an OTA upload. This time (Verifying...) progresses to (Uploading...) but freezes after the following output:
Over-The-Air upload request created successfully

Over-The-Air upload pending...

Initializing Over-The-Air update...

Starting Over-The-Air update attempt...

Downloading binary file...

I have tried adding various amounts of innocuous code but have been unable to suppress the warnings.

What is most strange is that one of the production systems produces the linker warnings but still accepts an OTA Upload. It would appear that there have been recent changes in this area as the output is now quite verbose when OTA upload is successful. I like that.

The other production system, the one looking after home security, is offline because ArduinoCloud.update() is hanging and I am unable to repair it via USB or OTA. This is a critical problem for me. Are there plans to fix the bug?

Regarding this problem with OTA uploads, I haven't done an investigation into it since @Human82's report indicated the problem was specific to uploads via the USB port. I don't know whether both types of upload failure have the same root cause or not.

However, I do know the OTA upload failure was reported to the Arduino Cloud developers by the support technician who handled the ticket you submitted via the contact form.

Same here! More information can vital to those of us who provide support to the Arduino community. This is especially important in the frustrating cases where we are unable to reproduce a problem reported by a user.

Yes. Work is already in progress on fixing the bug with the stderr handling.

As for your OTA upload problem, I have not been directly involved in that, but the developers are aware of the problem and I'm sure they will fix it ASAP.

I'll post an update here if I receive any news about a resolution of either problem from the developers.

Excellent! Thanks for the quick response.

I did some investigation into the OTA upload problem. I found that the bug with stderr output handling is specific to USB uploads. As you described, I found that there was no problem uploading sketches via the OTA port even when the compilation produced stderr output.

So I think your OTA upload problems must have a completely different cause.

Please post the sketch that was running on the board (not the sketch you were uploading) when you experienced this freeze. The freeze might be caused by a problem in your sketch code.

Thank you very much for your investigation and updates. As soon as the fix is deployed for IoT Things, I will test both USB and OTA uploads and report back.

If the OTA problem persists, I will provide a minimal test sketch to reproduce the problem. I believe I understand your request for the resident sketch at the time of upload, not the new sketch. It should be calling ArduinoCloud.update() as frequently as possible to permit the OTA update to initiate.

The production sketch is approximately 4000 lines long, so too large to post or be useful.

I fear the problem may be specific to my account, but there is no point in speculating on that until this fix is tested.

1 Like

Hi ptillisch,

I can upload via the USB port successfully now, thank you.

As you predicted, this did not solve the OTA issue. Would you like me to start a new Topic for this or continue with this one?

Great news! At least one of the problems is now resolved.

That is unfortunate.

It is definitely good to use a dedicated topic for the discussion of the OTA upload problem since we have now determined it is unrelated to the stderr handling bug. Since we already had some significant discussion on the OTA upload problem, I requested that a moderator split the existing discussion out to a dedicated topic, which has now been done (I'm replying in that topic now).

So we can proceed with the discussion here.

Thank you.

Here is the code that I can successfully upload via USB and is therefore the receiving code when attempting the OTA upload:

#include "thingProperties.h"

void setup() {
  // Initialize serial and wait for port to open:
  Serial.begin(9600);
  // This delay gives the chance to wait for a Serial Monitor without blocking if none is found
  delay(1500); 

  // Defined in thingProperties.h
  initProperties();

  // Connect to Arduino IoT Cloud
  ArduinoCloud.begin(ArduinoIoTPreferredConnection,false);
  
  setDebugMessageLevel(2);
  ArduinoCloud.printDebugInfo();
  
  pinMode(LED_BUILTIN, OUTPUT);

  test = 0;
}

void loop() {
  
  digitalWrite(LED_BUILTIN,HIGH);
  ArduinoCloud.update();
  digitalWrite(LED_BUILTIN,LOW);
  delay(100);
  test++;
}

The Thing has one integer variable "test" that is read-only and updates every 10 seconds.

Here is the output from the OTA attempt:

Over-The-Air upload available, compiling...

/usr/local/bin/arduino-cli compile --fqbn arduino:samd:nano_33_iot --build-cache-path /tmp --output-dir /tmp/3473073800/build --build-path /tmp/arduino-build-3F4A1CCB84645D2FFAAB2A952E78B786  --library /mnt/create-efs/webide/03/e1/03e1a6aeac7bb5900020bf52a73c0d25:rodgilmour/libraries_v2/Adafruit SSD1306 /tmp/3473073800/WDT_Test_apr12a
[info] Sketch uses 95828 bytes (36%) of program storage space. Maximum is 262144 bytes.
[info] Global variables use 5980 bytes (18%) of dynamic memory, leaving 26788 bytes for local variables. Maximum is 32768 bytes.

/home/builder/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/ld: warning: changing start of section .bss by 4 bytes
/home/builder/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/ld: warning: changing start of section .bss by 4 bytes
/home/builder/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/ld: warning: changing start of section .bss by 4 bytes
/home/builder/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/ld: warning: changing start of section .bss by 4 bytes
/home/builder/.arduino15/packages/arduino/tools/arm-none-eabi-gcc/7-2017q4/bin/../lib/gcc/arm-none-eabi/7.2.1/../../../../arm-none-eabi/bin/ld: warning: changing start of section .bss by 4 bytes


Over-The-Air upload request created successfully

Over-The-Air upload pending...

Initializing Over-The-Air update...

Downloading binary file...

Over-The-Air upload pending...

Initializing Over-The-Air update...

The flow is predictable and proceeds at a normal rate until it hits "Initializing Over-The-Air update...". It may freeze there or it may proceed to "Downloading binary file..." and freeze there. After a few minutes of being frozen, it might retry, as was captured above.

As you can see in the code, LED_BUILTIN is flashing to show calls to ArduinoCloud.update(). The LED becomes steady-on when "Initializing Over-The-Air update..." is reported so the Nano seems to be aware of the request.

If left long enough (say 10 minutes), it will sometimes give this output:

Board error received - OtaDownloadFail

It can be quite difficult to break out of this mode when frozen. It will stay pending even if I close the browser and open it again. Usually I can escape if I reset the Nano, wait a couple of minutes while it retries, then reset the Nano again. At this point I will usually get this output:
SHA256 mismatch - Wrong firmware detected after restart

I don't know if this is relevant, but two things occured around the time this started:

  1. I reset my account password
  2. I was testing Google Home integration.

Thanks for your help,

Rod

Hi @rodgilmour,

I believe the problem here is the delay(100) in the loop that makes the download phase of the OTA last longer than 10 minutes. Then a timeout occurs. Please try removing the delay(100) and replace it with the non blocking millis() check and try again.

Thank you for the response. I tested your suggestion but it still fails.

Today I received a new Nano 33 IoT. I went through the Online Device add process and this device can upload OTA just fine. So, at this point, it seems I am chasing a hardware or firmware issue.

For the record, the new Nano will successfully upload OTA even with the delay set to 1000. In fact, the output is more comprehensive, leading one to believe that some delay is expected.

As for the troublesome Nano 33 IoT, I deleted the device and added it back in, thus going through the firmware process again. This went quickly as it bypassed the firmware update process. This did not resolve the problem.

I then deleted the device, updated the firmware using IDE 2.3.2 and added it using the Online Device add. Still no luck with OTA upload.

The device accepts USB uploads and it communicates successfully with IoT Cloud so I am at a loss as to why it won't upload OTA. I have attempted everything I can think of. Any other suggestions on things to try?

If not, I will just write off the device and close the topic.

I see a similar post from ghu-01 under Web Editor. Maybe I won't give up on the device so quickly.

Arduino Nano 33 IOT Cloud upload issue

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.