Compilation and upload speed - what to expect?

Development Tools Uploading
1

Firstly: apologies if this post is in the wrong place - I could not see a more suitable area to post this question in.

My query is that I have noticed on videos showing code being compiled and uploaded, that it happens a lot quicker than when I do the same!

I'm using an Apple Air laptop (CPU:1.1 GHz Dual-Core Intel Core i3, Memory: 8 GB 3733 MHz LPDDR4X, IOS: Sequoia 15.3.1), and I'm using IDE version 2.3.4 and the boards I am using are marked: ESP32 -WROOM-32E.

The compilation time for the attached simple sketch is about 46 seconds.

The upload time (at 460800) is about 28 seconds.

Is this right? Are there any methods to speed up compilation time (apart from buying a more powerful computer....)?

/*
  Fade

  This example shows how to fade an LED on pin 9 using the analogWrite()
  function.

  The analogWrite() function uses PWM, so if you want to change the pin you're
  using, be sure to use another PWM capable pin. On most Arduino, the PWM pins
  are identified with a "~" sign, like ~3, ~5, ~6, ~9, ~10 and ~11.

  This example code is in the public domain.

  https://www.arduino.cc/en/Tutorial/BuiltInExamples/Fade
*/

int led = 9;         // the PWM pin the LED is attached to
int brightness = 0;  // how bright the LED is
int fadeAmount = 5;  // how many points to fade the LED by

// the setup routine runs once when you press reset:
void setup() {
  // declare pin 9 to be an output:
  pinMode(led, OUTPUT);
}

// the loop routine runs over and over again forever:
void loop() {
  // set the brightness of pin 9:
  analogWrite(led, brightness);

  // change the brightness for next time through the loop:
  brightness = brightness + fadeAmount;

  // reverse the direction of the fading at the ends of the fade:
  if (brightness <= 0 || brightness >= 255) {
    fadeAmount = -fadeAmount;
  }
  // wait for 30 milliseconds to see the dimming effect
  delay(30);
}
2

I'm not saying your selection of board processor is the issue or that the ESP32 Arduino tools are best avoided in my opinion, but you might try your sketch with a different board and see if there's a difference.

For example: compiling for an Uno R3 takes 7 seconds on a Raspberry Pi 4. And a Pi 4 is not exactly going to break any land speed records, but it serves my humble purposes.

Uploading takes 9 seconds.

My two cents.

3

Thanks for the comment: I just tried another board marked "ESP32-WROOM-32" with a USB-C port (the first board had a micro-USB port), and the figures are 31 seconds compilation (though I don't understand why a board would affect compile time? But clearly something was different from the first time), and 37 seconds upload.

I made sure I was doing nothing else with my laptop and closed all other apps down, whilst doing these timings.

With no board connected, compile time is consistently about 33 seconds.

I'd be interested to know what compile times is to be expected with this simple sketch, or whether there are simply too many environment variables to be able to make meaningful comparisons.

4

If the processor on the board doesn't change, the tool chain doesn't change. An ESP32 is an ESP32 is an ESP32. You'll get bloated code and slow compile times with any board that uses one.

5

HI - can you expand a bit on what you mean by this? Do you mean any ESP32 board?

6

Re-read the sentence you quoted. I don't feel that needs any expansion.

7

OK, sorry, maybe I'm having a senior moment, but your quote is over my head!!!

Thanks for your input though.

8

@van_der_decken meant that the ESP32 compiler is slow and the resulting compiled code is large and inefficient.

9

Did they compile for an ESP32 as well? Or did they use an Uno or Mega or ...

If you enable Show verbose output during compilation under file/preferences, you can see how much more is processed if you compile for a ESP32 based board compared to e.g. an Uno. You only need to compile to see that, no need for upload.

A second compile of the same code should be faster as the compiled cide is cached.

Before your code is compiled, there is a process called the Arduino builder that determines what all needs to be compiled; with the complexity of the ESP32 core, that process takes long.

And lastly a virus scanner can slow the process down.

I can not comment on the upload speed, that is determined by the size of the hex/bin file.

10

A quirk of the Arduino IDE is that after hitting the ‘upload’ button it will start with compiling again. As said before, a second compilation will use previously compiled code, but still…

11

Is that the 2020 model? Seems like all the Core i* in Macs have hyperthreading, so 4 threads.

Last year's topic: Excessive time for simple build

Last month's topic -- Funded project to speed compile time on Mac -- mentions a 10-second overhead on Mac with Apple silicon

I dug out the 2018 Air, dual-core 4-thread i5

--clean cached
prepare 8.891 9.336
detecting libraries 0.761 1.267
function prototypes 0.831 0.869
compiling sketch 0.840 0.866
compiling libraries 0.001 0.001
compiling core 14.799 0.107
linking 2.160 2.162
final 17.132 16.277
--------
Total 45.413 30.886

The overhead totals 26 seconds. The last command that takes several seconds is /esp32/tools/esptool_py/4.6/esptool --chip esp32 merge_bin

So my time matches yours. More than half(!) is the pre- and post-processing.

12

Thanks so much for the comprehensive replies. I'm very new to this, and clearly it's a topic that has had a lot of attention and, as suspected, the answers are complex and nuanced.

To answer kenb4; yes, my laptop is a 2020 model. Thanks for doing the comparison - it's good to know that it's not just me doing, or not doing, something stupid!

Out of interest, I turned off my Trusteer Rapport End-Point protection, and the results were 39 seconds to compile, and 30 seconds to upload. So slightly faster, but could that be due to sequential compilations and caching? But certainly not significantly faster. Perhaps a much larger, more complex sketch would expose larger compilation time effects of having Trusteer turned off or on.

As a sanity check, I turned Trusteer back on again and got the same numbers as before in my original post, just slightly faster again - I now know that that is most likely because the compiler caches stuff.

Finally, I went off line (with Trusteer still running), and checked that I got the same results as when on-line; which I did.

Also good to know that you can't Upload without compiling again. I thought it was just me not finding the right setting. Very irritating!

So I shan't be rushing out to buy a better laptop or Apple M2 Max anytime soon...

1 Like
13

And given that 'compiling core' is mostly eliminated on second and subsequent runs, the overhead % is then even worse ...

14

After linking, esptool runs twice. On Linux, it's a Python script, python3 /long/path/esptool.py. But on Mac, it's an executable.

$ time ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool --chip esp32 elf2image --flash_mode dio --flash_freq 80m --flash_size 4MB --elf-sha256-offset 0xb0 -o /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bin /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.elf
esptool.py v4.6
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.

real	0m7.646s
user	0m0.161s
sys 	0m0.120s

0.28 seconds doing work, and over 7 seconds doing... nothing. It's run twice, once for elf2image, and then again for merge_bin

$ time ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool --chip esp32 merge_bin -o /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.merged.bin --fill-flash-size 4MB --flash_mode keep --flash_freq keep --flash_size keep 0x1000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bootloader.bin 0x8000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.partitions.bin 0xe000 ~/Library/Arduino15/packages/esp32/hardware/esp32/3.0.5/tools/partitions/boot_app0.bin 0x10000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bin
esptool.py v4.6
Wrote 0x400000 bytes to file /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.merged.bin, ready to flash to offset 0x0

real	0m7.609s
user	0m0.137s
sys 	0m0.124s

totaling a half-second of work and almost 15 seconds of nothing. It takes the same amount of time to print the usage

$ time ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool
esptool.py v4.6
usage: esptool [-h]
...
  --connect-attempts CONNECT_ATTEMPTS
                        Number of attempts to connect, negative or 0 for infinite. Default: 7.

real	0m7.695s
user	0m0.144s
sys 	0m0.128s

Getting the plain script to run wasn't too difficult

cd ~/dev
git clone git@github.com:espressif/esptool.git
cd esptool/
git checkout v4.6
python3 -m pip install pyserial

You may need to install pyserial (one reason it is distributed as an executable?)

$ time python3 ~/dev/esptool/esptool.py --chip esp32 elf2image --flash_mode dio --flash_freq 80m --flash_size 4MB --elf-sha256-offset 0xb0 -o /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bin /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.elf
esptool.py v4.6
Creating esp32 image...
Merged 2 ELF sections
Successfully created esp32 image.

real	0m0.155s
user	0m0.114s
sys 	0m0.032s
$ time python3 ~/dev/esptool/esptool.py  --chip esp32 merge_bin -o /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.merged.bin --fill-flash-size 4MB --flash_mode keep --flash_freq keep --flash_size keep 0x1000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bootloader.bin 0x8000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.partitions.bin 0xe000 ~/Library/Arduino15/packages/esp32/hardware/esp32/3.0.5/tools/partitions/boot_app0.bin 0x10000 /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.bin
esptool.py v4.6
Wrote 0x400000 bytes to file /private/var/folders/x_/yv8yw5dj7qdg7vw05b8xzgtw0000gn/T/arduino/sketches/FD07796727C8EF11316CF0BBA2DD3C53/t1363459.ino.merged.bin, ready to flash to offset 0x0

real	0m0.125s
user	0m0.084s
sys 	0m0.031s

esptool also runs once for elf2image at the beginning prepare stage, making that slow too. I have it added it under Settings | Privacy & Security | Developer Tools. No noticeable effect.

$ file ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool 
/Users/kenb4/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool: Mach-O 64-bit executable x86_64
$ ls -l@ ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool 
-rwxr-xr-x@ 1 kenb4  staff  5971680 Oct 15 14:07 /Users/kenb4/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/esptool
	com.apple.provenance	     11

There is some talk on the internet that running an executable the first time on a Mac is slow, but this is slow every time. The provenance attribute may indicate it has cleared quarantine.

How about replacing the executable with a shell script to run the Python script?

cd ~/Library/Arduino15/packages/esp32/tools/esptool_py/4.6/
mv esptool esptool.x86
printf '#!/bin/bash\npython3 ~/dev/esptool/esptool.py "$@"\n' > esptool.sh
chmod a+x esptool.sh
ln -s esptool.sh esptool
--clean cached
prepare 0.952 0.954
detecting libraries 0.714 0.823
function prototypes 0.797 0.838
compiling sketch 0.850 0.864
compiling libraries 0.001 0.001
compiling core 15.046 0.119
linking 2.169 2.062
final 0.619 0.669
--------
Total 21.148 6.330

Much better. BTW, the upload (for a different almost-empty sketch) was

Changing baud rate to 1500000
...
Wrote 279184 bytes (156282 compressed) at 0x00010000 in 2.8 seconds (effective 796.5 kbit/s)...
15

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.