Random upload fails, partial progress

I'm on Fedora 39 (not a vm), and having random errors uploading to a Nano ESP32.

When this happens, if I put the device into bootloader mode by double-pressing the reset button, and getting the green glowing led, I can get one successful upload done, but then any subsequent ones fail either at 0%, or somewhere before 100%.

Occasionally I get another full upload, but that's very rare.

I'm on v2.2.1 of the IDE.

I have the following udev rule which I dug into and realised without this, I couldn't even get dfu-util access. The most important part is the 0666 mode, even if I'm in the group of the /dev/ttyAMC, the dfu-util couldn't access without sudo.

SUBSYSTEMS=="usb", ATTRS{idVendor}=="2341", MODE="0666"

This is typically what I see AFTER the first good install.

Opening DFU capable USB device...
Device ID 2341:0070
Device DFU version 0101
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 0101
Device returned transfer size 4096
Copying data from PC to DFU device

Download	[                         ]   0%            0 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

Or...

Download	[                         ]   0%            0 bytes
Download	[=                        ]   5%        16384 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

Let me know what other info you need to help debug.

I have 2 Nano ESP32 boards, which should be identical, but the lights are a bit confusing from what I've read, so some detail on this would be great.

Board 1

When plugged in, I get a constant green led next to the USB-C socket.
Double pressing the reset button starts the main onboard led flashing Blue red green in sequence.
Pressing the reset button again gets a glowing green led.

Can someone explain what these all mean? What "mode" is it in when it's flashing blue/red/green?

Board 2

When plugged in I get a constant green led on one side of the usbc socket, but also a constant yellow led on the other side.


I'm also intrigued to know why dfu is being used if a serial port/device has been created by the OS? And/or if it can't be used, what exactly has gone "wrong" to prevent it? And can I add anything to my own code to help?

Do you enable read write access to /dev/ttyAMC?

Hello @robsco, sorry to hear that! Please update both boards using this procedure and let me know if they now match. Note that some early production models used an LED with blue and green reversed, so maybe that is not helping as well. :wink:

Entering the bootloader should be easy to do, once you know how it's working: one reset click will start the RGB colors and then your sketch; if you click reset a second time, while the colors are shown, the board will enter DFU bootloader mode.

That serial port is a fully software defined UART, created by the TinyUSB stack on the ESP32. Using the software-controlled USB instead of the hardware one enables sketch customizations of the USB interface, but at the cost of losing the hardware firmware update that's baked in the ESP32 chips. DFU should replace that process (when it works! :man_facepalming:) since it is supported by TinyUSB and nicely integrates with the Arduino IDE.

Not sure. That udev rule will allow any user access to any Arduino board on your PC, so it's more than what was required, but it's obviously OK.
It seems that either the board disconnects or another program interferes with the DFU upload. I assume changing USB ports, cabling etc doesn't matter?

The yellow light on the other side of the usbc sockets turned out to be pin D13 being grounded on the foam pad it was sat on - when I took it out and touched it with my finger it turned yellow on both boards.

Bit strange as I don't utilise that pin in my code, but hey ho, I'll ensure they're always in a breadboard from now on, which now removes the yellow led on the other side of the usb.

Both boards were fully reset as per the link provided, all steps followed and results as expected.

Attempting a normal upload after resetting the boards failed again...

Opening DFU capable USB device...
Device ID 2341:0070
Device DFU version 0101
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 0101
Device returned transfer size 4096
Copying data from PC to DFU device

Download    [                         ]   0%            0 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

Pressing the reset button once, and waiting for the colour sequence, then pressing again, resulting in the green glow...

Upload succeeds.

Trying to upload again without being in bootloader mode gives the same failure as before.

Opening DFU capable USB device...
Device ID 2341:0070
Device DFU version 0101
Claiming USB DFU Interface...
Setting Alternate Interface #0 ...
Determining device status...
DFU state(2) = dfuIDLE, status(0) = No error condition is present
DFU mode device DFU version 0101
Device returned transfer size 4096
Copying data from PC to DFU device

Download    [                         ]   0%            0 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

So it seems the only way to upload my code is to ALWAYS go into bootloader mode.

@lburelli What else can I try now?

I have tried lots of different cables and different ports.

I also tried the other USB Mode from the Tools menu, Debug mode (Hardware CDC), but got the same result of failing.

What is actually different to the upload process when in bootloader mode? Is it a different process/method to do the upload?

$ dfu-util -l
dfu-util 0.11

Copyright 2005-2009 Weston Schmidt, Harald Welte and OpenMoko Inc.
Copyright 2010-2021 Tormod Volden and Stefan Schmidt
This program is Free Software and has ABSOLUTELY NO WARRANTY
Please report bugs to http://sourceforge.net/p/dfu-util/tickets/

Found DFU: [2341:0070] ver=0100, devnum=85, cfg=1, intf=0, path="3-1.2", alt=0, name="Arduino DFU", serial="4827E2FD7ED8"

$ ll /dev/serial/by-id/
total 0
lrwxrwxrwx 1 root root 13 Dec 27 17:25 usb-Arduino_Nano_ESP32_4827E2FD7ED8-if01 -> ../../ttyACM1
lrwxrwxrwx 1 root root 13 Dec 26 23:56 usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2498898-if00 -> ../../ttyACM0

$ ll /dev/ttyACM1 
crw-rw-rw- 1 root dialout 166, 1 Dec 27 17:25 /dev/ttyACM1

Thanks for the very detailed explanation (and happy new year!).

That is not a good thing... :cold_face:
I suggest you don't place any powered electronics on that mat, since it is clearly conducting and may silently interfere with other important circuit nets. That said,

The only difference between these tests should be the sketch that you are trying to load (the "bootloader" is in fact a permanently available, quite empty sketch :wink:). Can you try with the "Blink" example?

@lburelli thanks for bearing with me, okay so it seems like it's my sketch causing it, once it gets installed and running, messes up any subsequent uploads.

Using the following blink example...

https://docs.arduino.cc/built-in-examples/basics/Blink

PC has been rebooted since plugging in anything Arduino related.

First upload fails, not in bootloader mode, and my sketch is running on it.

Download	[                         ]   0%            0 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

Going into bootloader mode, works...

Download	[                         ]   0%            0 bytes
Download	[=                        ]   4%        12288 bytes
Download	[==                       ]   8%        24576 bytes
Download	[===                      ]  12%        36864 bytes
Download	[====                     ]  16%        49152 bytes
Download	[=====                    ]  21%        61440 bytes
Download	[======                   ]  25%        73728 bytes
Download	[=======                  ]  28%        81920 bytes
Download	[========                 ]  32%        94208 bytes
Download	[=========                ]  36%       106496 bytes
Download	[==========               ]  40%       118784 bytes
Download	[===========              ]  45%       131072 bytes
Download	[===========              ]  46%       135168 bytes
Download	[============             ]  49%       143360 bytes
Download	[=============            ]  52%       151552 bytes
Download	[==============           ]  56%       163840 bytes
Download	[===============          ]  60%       176128 bytes
Download	[================         ]  64%       188416 bytes
Download	[=================        ]  69%       200704 bytes
Download	[==================       ]  73%       212992 bytes
Download	[===================      ]  76%       221184 bytes
Download	[====================     ]  80%       233472 bytes
Download	[=====================    ]  84%       245760 bytes
Download	[======================   ]  88%       258048 bytes
Download	[=======================  ]  93%       270336 bytes
Download	[======================== ]  97%       282624 bytes
Download	[=========================] 100%       286192 bytes
Download done.
DFU state(7) = dfuMANIFEST, status(0) = No error condition is present
DFU state(2) = dfuIDLE, status(0) = No error condition is present
Done!

Yellow onboard LED blinks.

Touching nothing and just trying another upload straight away, works, woohoo.

Several repeated uploads all upload fine.

Switching back to my sketch, first one uploads okay (as the blink is installed on it), but then subsequent ones fail.

Trying to upload the blink again (but with my sketch running)...

Download	[                         ]   0%            0 bytes
Download	[                         ]   2%         8192 bytes
Download	[=                        ]   4%        12288 bytes
Download	[==                       ]   8%        24576 bytes
Download	[===                      ]  12%        36864 bytes
Download	[====                     ]  16%        49152 bytes
Download	[=====                    ]  21%        61440 bytesdfu-util: Error during download get_status (LIBUSB_ERROR_PIPE)
Failed uploading: uploading error: exit status 74

Going back to bootloader, blink uploads fine.

Then all subsequent ones work fine.

So now it's down to debugging my code, and most importantly figuring out the WHY.

The code controls 4 separate LED strips, showing various flame effects.

I have 2 pots to control brightness and which effect/palette to display, plus an LED to light if the brightness would result in more than 3A being pulled from the power supply.

I'll include the full sketch, with some commented out tests, etc.

Any advice welcome...

#include <Arduino.h>
#include <FastLED.h>
#include <math.h>

#define NUM_LEDS  51

#define COOLING  50
#define SPARKING 60

#define FPS 30   // 60 is ideal

#define ARM_1_PIN 2
#define ARM_2_PIN 3
#define ARM_3_PIN 4
#define ARM_4_PIN 5

#define VOLT_LIMIT 5
#define MILLIAMP_LIMIT 3000

#define AMP_LIMIT_LED_PIN 7

#define MIN_BRIGHTNESS 10
#define MAX_BRIGHTNESS 255

#define BRIGHTNESS_PIN A4

#define PALETTE_PIN A2

#define NUM_AVERAGE_SAMPLES 30

byte heat[ 4 ][ NUM_LEDS ] = { 0 };

CRGB leds[ 4 ][ NUM_LEDS ] = { 0 };

void setup()
{
    delay( 2000 ); // power-up safety delay

    FastLED.addLeds<WS2812B, ARM_1_PIN, GRB>( leds[ 0 ], NUM_LEDS );
    FastLED.addLeds<WS2812B, ARM_2_PIN, GRB>( leds[ 1 ], NUM_LEDS );
    FastLED.addLeds<WS2812B, ARM_3_PIN, GRB>( leds[ 2 ], NUM_LEDS );
    FastLED.addLeds<WS2812B, ARM_4_PIN, GRB>( leds[ 3 ], NUM_LEDS );

    pinMode( AMP_LIMIT_LED_PIN, OUTPUT );   // just in case it's not set

    set_max_power_indicator_LED( AMP_LIMIT_LED_PIN );
    
    FastLED.setMaxPowerInVoltsAndMilliamps( VOLT_LIMIT, MILLIAMP_LIMIT );
        
    FastLED.setBrightness( MAX_BRIGHTNESS );

    Serial.begin( 115200 );

    if ( 0 )
    {
        all_white( leds[ 0 ] );
        all_white( leds[ 1 ] );
        all_white( leds[ 2 ] );
        all_white( leds[ 3 ] );

        FastLED.show();
    }
}

CRGBPalette16 palettes[] = {
    HeatColors_p,
    RainbowColors_p,
    CloudColors_p,
    ForestColors_p,
    OceanColors_p
};

void loop()
{
    EVERY_N_MILLISECONDS( 1000 / FPS ){
    
        // update the brightness

        FastLED.setBrightness( _get_normalised_brightness_value() );

        for ( int i = 0; i < 4; i++ )
        {
            fire( leds[ i ], heat[ i ] );

            // knight_rider( leds[ i ] );

             // all_white( leds[ 0 ] );
             // all_white( leds[ 1 ] );
             // all_white( leds[ 2 ] );
             // all_white( leds[ 3 ] );
        }
  
        FastLED.show();
    }
}

int _get_normalised_brightness_value()
{
    int brightness_pin_value = analogRead( BRIGHTNESS_PIN );

    int brightness_value = map(
        brightness_pin_value,
        0, 1023,
        MIN_BRIGHTNESS, MAX_BRIGHTNESS
    );

    return brightness_value;
}


void fire( CRGB leds[], byte heat[] )
{
    // cool down every cell a little
    
    for ( int i = 0; i < NUM_LEDS; i++ )
    {
        heat[ i ] = qsub8( heat[ i ], random8( 0, ( ( COOLING * 10 ) / NUM_LEDS ) + 2 ) );
    }
  
    // heat from each cell drifts 'up' and diffuses a little

    for ( int i = NUM_LEDS - 1; i >= 2; i-- )
    {
        heat[ i ] = ( heat[ i - 1 ] + heat[ i - 2 ] + heat[ i - 2 ] ) / 3;
    }
    
    // randomly ignite new 'sparks' of heat near the bottom

    if ( random8() < SPARKING )
    {
        int y = random8( 7 );

        heat[ y ] = qadd8( heat[ y ], random8( 160, 255 ) );
    }

    // map from heat cells to LED colors

    CRGBPalette16 palette = HeatColors_p;   // default

    int num_palettes = sizeof( palettes ) / sizeof( CRGBPalette16 );

    // map the analogue input (0-1023) to an index in our array of palettes

    int analogue_value = analogRead( PALETTE_PIN );

    int averaged_analogue_value = _averageAnalogueRead( PALETTE_PIN );

    int mapped_value = map(
        averaged_analogue_value,
        0, 1023,
        0, num_palettes - 1
    );

    char serial_out[255];

    sprintf( serial_out, "Total palettes %d, analogue pin %d, averaged analogue value %d, mapped %d", num_palettes, analogue_value, averaged_analogue_value, mapped_value );



    // Serial.println( serial_out );

    palette = palettes[ mapped_value ];

    for ( int i = 0; i < NUM_LEDS; i++ )
    {
        // Scale the heat value from 0-255 down to 0-240
        // for best results with color palettes.

        byte colorindex = scale8( heat[ i ], 240 );

        CRGB color = ColorFromPalette( palette, colorindex );

        leds[ i ] = color;
    }
}

int _averageAnalogueRead( int pin )
{
    static int values[ NUM_AVERAGE_SAMPLES ] = { 0 };

    static int last_element = -1;

    last_element ++;

    if ( last_element == NUM_AVERAGE_SAMPLES )
    {
        last_element = 0;
    }

    int value = analogRead( pin );

    values[ last_element ] = value;

    int average = 0;

    for ( int i = 0; i < NUM_AVERAGE_SAMPLES; i++ )
    {
        average += values[ i ];
    }

    return average / NUM_AVERAGE_SAMPLES;
}

void all_white( CRGB leds[] )
{
    fill_solid( leds, NUM_LEDS, CRGB::White );
}

void knight_rider( CRGB leds[] )
{
    uint8_t sinBeat = beatsin8( 120, 0, NUM_LEDS - 1, 0, 0 );

    leds[ sinBeat ] = CRGB::Red;

    fadeToBlackBy(leds, NUM_LEDS, 10);
}

I'd love to know what could be causing this behaviour.

1 Like

Glad we found out the issue, and thanks for posting the code!

Since the upload is happening in the background, while the sketch is running, there might be a timeout in the software USB stack caused by heavy CPU use by the user code.

I have seen other reports that excessive Serial.print() caused this behavior, but this doesn't seem to be the issue here. However I see a different subtle issue in your code - a loop with no delays. The EVERY_N_MILLISECONDS macro basically amounts to an if (false) most of the time, and this will cause loop() to be run at unholy speeds, stealing CPU cycles from the background USB task.

I think adding a simple delay(1) in your loop() should fix the issue with no noticeable effect. Let me know if this works for you!

@lburelli That makes a lot of sense.

However, even with a delay(100) it still doesn't want to upload on a subsequent attempt.

Uploading in bootloader works first time (as before), but no joy after that.

void loop()
{
    EVERY_N_MILLISECONDS( 1000 / FPS ){
    
        // update the brightness

        FastLED.setBrightness( _get_normalised_brightness_value() );

        for ( int i = 0; i < 4; i++ )
        {
            fire( leds[ i ], heat[ i ] );

            // knight_rider( leds[ i ] );

             // all_white( leds[ 0 ] );
             // all_white( leds[ 1 ] );
             // all_white( leds[ 2 ] );
             // all_white( leds[ 3 ] );
        }
  
        FastLED.show();
    }

    delay(100);
}

Hmm. Sorry but I don't have a lot more to try, honestly... :confused:
To debug further, does the situation improve removing parts of the sketch code?

I assume the issue goes away with commenting the FastLED.show() (and keeping some delay in the loop)? If so, there might be something in the FastLED implementation that messes with the Nano ESP32 firmware download, but I will need to find time to investigate more.

@lburelli Thanks, I'll start to strip it down until I get to a working version.

Removing the FastLED.show() does indeed "fix it".

Or cutting down my code to only operate 2 arrays of LEDs, rather than 4, seems to make it not break too.

    FastLED.addLeds<WS2812B, ARM_1_PIN, GRB>( leds[ 0 ], NUM_LEDS );
    FastLED.addLeds<WS2812B, ARM_2_PIN, GRB>( leds[ 1 ], NUM_LEDS );
    // FastLED.addLeds<WS2812B, ARM_3_PIN, GRB>( leds[ 2 ], NUM_LEDS );
    // FastLED.addLeds<WS2812B, ARM_4_PIN, GRB>( leds[ 3 ], NUM_LEDS );

I do see this from the FastLED library when compiling, not sure if it has an effect, or is something I can "do" to make it go way?

In file included from /home/rob/Arduino/libraries/FastLED/src/FastLED.h:75,
                 from /home/rob/Arduino/2023-12-17-04-array-of-arrays-single-function/lamp/lamp.ino:2:
/home/rob/Arduino/libraries/FastLED/src/fastspi.h:157:23: note: #pragma message: No hardware SPI pins defined.  All SPI access will default to bitbanged output
 #      pragma message "No hardware SPI pins defined.  All SPI access will default to bitbanged output"
                       ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

btw, after some more research into the bootloader and how the controller gets programmed, I have a couple of questions...

Am I correct in thinking that the bootloader is simply a sketch that waits for commands, to program the flash? That would make sense, when I actively put the Arduino into bootloader mode, it simplifies what the controller is doing when I try to program it - and why it works everytime that way.

If that's true, is the bootloader code also somehow injected into my sketch, so the controller regularly checks for incoming updates? Or does the upload process from the IDE quickly reset the device, (into bootloader mode?) then quickly provides the new flash/program?

It's this part of re-flashing the controller while MY code is running, that's got me stumped.

Apart from that I think I understand most of it.

 #      pragma message "No hardware SPI pins defined.  All SPI access will default to bitbanged output"

I will check what that message means, it should not print something like that and it might be the cause (if it's disabling interrupts, for example).

By the way, you are using "By GPIO number (legacy)" in the "Tools" / "Pin Numbering" menu, right? If not, sorry for not mentioning this before - please read this guide on the subject!

Re/ the upload process: a copy of the DFU code is included in every sketch uploaded and is always active in the background. When a double tap is detected, instead of the user program, a pre-built sketch stored in a different Flash section is started, and that is an empty sketch which basically just fades the green LED and waits for DFU. You can find the source here if you're interested!

@lburelli yes I'm using the GPIO pin numbering method, due to using FastLED.

I'm curious as to whether, and how, this could cause problems having this set incorrectly? Would things simply not "work" due to the incorrect pins being activated or read from? Or could it cause significant crashes or worse problems?

I was under the impression the tap (to reset) then the following tap to get into bootloader mode was a feature of the chip? Where can I read about the internals of all this behaviour? I'm very keen to read about and learn the underlying internals to know "why" things are happening.

I don't suppose you'll have chance to try and re-create my issue? At the very least to confirm it's a problem not just with my setup/hardware, etc.?