Why is the compiler so slow?

It's currently taking me 5 minutes to compile a sketch (<1K lines) using the Arduino IDE. I'm very new to Arduino but I just can't picture what the compiler is doing for all this time. Does anyone know? For comparison it takes about 2 seconds to compile my Java chess program (15K lines). Just as an amateur, I've even written compilers in high level languages and even these were blazingly fast in comparison.

Does anyone know what it's doing for that time? SETI? Protein folding at home?

Likely a conflict with your antivirus program.

1 Like

Or could be a problem with whatever machine you are using the Arduino IDE on.

Turning off AVG did improve things a bit. Still not exactly fast though :wink: Is there a way of allowing the Arduino IDE to be trusted?

My own PC is fairly fast and only one core reaches 20% during the compilation. The rest are flatlining. It doesn't connect to the Arduino (a ttgo t-watch 2020) until a number of minutes have gone past. Passing the data across seemed performant.

until a number of minutes have gone past.

I've never seen anything like that. It takes less than one minute, from pushing the "upload" button, until Arduino startup. Even for quite large programs. Windoze 10, on a eight year old, 2.9 GHz Dell computer with four cores.

Just for Grins, try Linux Mint 20 and see if it works better. If you have an old machine even the better. You can install it along side windows or simply run it in demo mode. I have several sketches that are well over 1K lines long and they compile and uploaded in maybe 10 - 15 seconds. If I go to the ESP stuff it takes about twice as long. Note the first compile with a program takes longer, I believe the compiler copies everything into a single temporary file then does its thing. If your disk is slow or fragmented it will take forever. With linux after installing arduino you will be prompted to add your name to dial out then reboot. After that it works great. I ran dual boot for several years not only one of my machine has windows.

mike_bike_kite:
My own PC is fairly fast and only one core reaches 20% during the compilation.

Maybe tweak the jvm options? Not sure where the config file is for that, but maybe extra memory, make sure debugging is off - that kind of thing.

There is a file named arduino.l4j.ini in the Arduino IDE installation folder:

Arduino/build/shared/revisions.txt at 1.8.13 · arduino/Arduino · GitHub

  • Windows: JVM memory settings can be customized editing arduino.l4j.ini and arduino_debug.l4j.ini files

mike_bike_kite:
Turning off AVG did improve things a bit. Still not exactly fast though :wink: Is there a way of allowing the Arduino IDE to be trusted?

My own PC is fairly fast and only one core reaches 20% during the compilation. The rest are flatlining. It doesn't connect to the Arduino (a ttgo t-watch 2020) until a number of minutes have gone past. Passing the data across seemed performant.

Arduino IDE only uses one core, it was never coded to support multiple cores and there's little reason, the majority of codes are small the compiler usually completes in some seconds. The largest code I ever created and compiled was 26k final size.

What board are you using?
Some boards (ESPxxx, NRFxxxx) have a large amount of OS files to compile as well as the user code.

Turn on "verbose" for compilation in the preferences panel, and it might show more details.

1 Like

It's an ESP32

Chipset ESP32, dual-core MCU, PMU power management

wilykat:
Arduino IDE only uses one core, it was never coded to support multiple cores and there's little reason, the majority of codes are small the compiler usually completes in some seconds. The largest code I ever created and compiled was 26k final size.

I just tried compiling an ESP32 example using Arduino IDE 1.8.9 on Ubuntu 20.04, and it used 100% on all 8 threads while compiling for the first time. Subsequent builds are much faster, because the core libraries will be cached, so only your sketch has to be rebuilt (this cannot be parallelized).

gilshultz:
Just for Grins, try Linux Mint 20 and see if it works better. If you have an old machine even the better.

I concur.
Compilation of large libraries (especially ESP8266/ESP32 because of the large core libraries) seems to be significantly faster on Linux (at least it was a couple of years ago when I still used Windows from time to time). I did a test in this thread, and Linux was 2-3 times faster than Windows (I couldn't find an explanation, so if anyone has an idea?):

Ubuntu 16.04 Windows 10
Initial compilation: 11.5s 23.8s
Normal compilation: 1.8s 5.9s

You can boot Ubuntu or Mint from a USB stick and try it out, to see if it's faster. You can then install it alongside Windows.

Pieter

wilykat:
Arduino IDE only uses one core, it was never coded to support multiple cores

Recent IDE versions have support for parallel compilations using multiple cores:

wilykat:
there's little reason, the majority of codes are small the compiler usually completes in some seconds. The largest code I ever created and compiled was 26k final size.

I guess you haven't tried machine learning on the Nano 33 BLE or Portenta H7 boards. That Mbed OS is really a beast to compile.

The days of Arduino == AVR are long past.

1 Like

I believe the compiler is pulling each dependant file separately from an external sources which then kicks off the virus checker on each occasion. If this happens for each library recursively then that might explain the 5 min compile time. I don't think any sane person would turn off their anti virus program as a solution though.

Could the compiler be changed so it just uses a local cached version of a file if available? Perhaps having different cache time outs depending on how new the downloaded file is ie if a library hasn't been touched in years then there's no need to reload it for at least a week. You could also have a low priority background task that checks for new versions of files you often use and pull these into your cache area. This could change the compile times from 5 min to just a few sec. I don't believe it would be a lot of work to implement.

@PieterP - Linux will be faster as it doesn't need the virus checker but I don't think I'm prepared to abandon all the other software I currently use on Win 10. I am however quite familiar with shell programming etc and even still use vim as my main editor.

1 Like

PieterP:
and Linux was 2-3 times faster than Windows (I couldn't find an explanation, so if anyone has an idea?)

No-one in the Linux world would be the least bit surprized by this. But this isn't the place...

Did you exclude all Arduino folders in your anti virus program?

  • Your Arduino installation folder (C:\Program Files (x86)\Arduino)
  • Arduino15 folder (C:\Users\YOUR_USERNAME\AppData\Local\Arduino15)
  • Arduino libraries and projects folder (C:\Users\YOUR_USERNAME\Documents\Arduino)

Try excluding these folders from Windows Defender too.

Edit: It might not work however. My git is very slow on Windows 10 for no apparent reason. When I run git in a Windows 7 virtual machine inside the Windows 10 it's blazing fast. But on the Windows 10 machine it takes annoyingly long times to do even a small commit on a small project. And I haven't been able to find a solution.

1 Like

mike_bike_kite:
Turning off AVG did improve things a bit. Still not exactly fast though :wink: Is there a way of allowing the Arduino IDE to be trusted?

My own PC is fairly fast and only one core reaches 20% during the compilation. The rest are flatlining. It doesn't connect to the Arduino (a ttgo t-watch 2020) until a number of minutes have gone past. Passing the data across seemed performant.

In other words the machine is thrashing somehow? Would agree with JVM starved of heap space is a possible
cause, or lack of RAM, or lack of virtual memory for JVM to grow into? Is the machine solid state disk?

mike_bike_kite:
@PieterP - Linux will be faster as it doesn't need the virus checker but I don't think I'm prepared to abandon all the other software I currently use on Win 10. I am however quite familiar with shell programming etc and even still use vim as my main editor.

You don't have to give it up. When programming Arduino, I usually only need the Arduino IDE, VSCode and a web browser, so in that case you can easily dual-boot and use Linux just for Arduino, while using Windows for everything else. Whether that's the right approach for you depends on your preferences of course (and on how much SSD space you have available).
If you can't figure out why Windows is so slow, and if it keeps bothering you, it might be worth a try.

MarkT:
No-one in the Linux world would be the least bit surprized by this.

I'm not surprised that GCC is faster on Linux, but 3 times faster is a huge difference.

With the latest Windows 10, you can install linux under WSL2 and use arduino cli to compile. Have not tried doing an upload that way, but it did work to compile marlin firmware.

It ran fairly quick when I first ran it today with the verbose flag turned on. Looking at the messages as they whizzed by, it suggested it was getting cached versions of previous compilations - I don't recall the exact message.

For my own sanity I tried compiling again an hour or two later and it's back to taking a number of minutes. I've attached the verbose output of the compilation - it's 4.5MB long ????? It seems to endlessly repeat the directory names which makes the output unreadable - single lines are 8500 characters long!! There's verbose and there's verbal diarrhoea. It turns out I got an error when trying to attach the file so I'm guessing the file is too large - nice one! Why don't they use some form of short form for the directory names like %SDK%. I've put an example line at the bottom.

Weirdly, it does save space by not showing any timings so I've no idea what part is taking all the time. It felt like everything was running fairly slow.

David - I already have Linux on my machine as well as WIn 10 but it seems silly to use a different OS just to run Arduino IDE. 2% of users use Linux while 90% use Windows - it would seem wise to get the compiler to run properly on Windows. I also have plenty of other compilers on my desktop that run fine under Windows so I'm guessing it's an Arduino issue.

Example line (I had to remove some of the line to allow me to post it):
"C:\Users\Mike\AppData\Local\Arduino15\packages\esp32\tools\xtensa-esp32-elf-gcc\1.22.0-80-g6c4433a-5.2.0/bin/xtensa-esp32-elf-g++" -DESP_PLATFORM "-DMBEDTLS_CONFIG_FILE="mbedtls/esp_config.h"" -DHAVE_CONFIG_H -DGCC_NOT_5_2_0=0 -DWITH_POSIX "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/config" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/app_trace" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/app_update" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/asio" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/bootloader_support" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/bt" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/coap" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/console" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/driver" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp-tls" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp32" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp_adc_cal" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp_event" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp_http_client" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp_http_server" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/libsodium" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/log" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/lwip" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/mbedtls" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/mdns" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/micro-ecc" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/mqtt" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/newlib" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/nghttp" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/nvs_flash" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/openssl" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/protobuf-c" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/protocomm" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/pthread" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/sdmmc" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/smartconfig_ack" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/soc" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/spi_flash" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/spiffs" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/tcp_transport" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/tcpip_adapter" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/ulp" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/vfs" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/wear_levelling" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/wifi_provisioning" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/wpa_supplicant" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/xtensa-debug-module" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp-face" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp32-camera" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/esp-face" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4/tools/sdk/include/fb_gfx" -std=gnu++11 -Os -g3 -Wpointer-arith -fexceptions -fstack-protector -ffunction-sections -fdata-sections -fstrict-volatile-bitfields -mlongcalls -nostdlib -w -Wno-error=maybe-uninitialized -Wno-error=unused-function -Wno-error=unused-but-set-variable -Wno-error=unused-variable -Wno-error=deprecated-declarations -Wno-unused-parameter -Wno-unused-but-set-parameter -Wno-missing-field-initializers -Wno-sign-compare -fno-rtti -c -w -x c++ -E -CC -DF_CPU=240000000L -DARDUINO=10813 -DARDUINO_T-Watch -DARDUINO_ARCH_ESP32 "-DARDUINO_BOARD="T-Watch"" "-DARDUINO_VARIANT="twatch"" -DESP32 -DCORE_DEBUG_LEVEL=4 -DBOARD_HAS_PSRAM -mfix-esp32-psram-cache-issue "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\cores\esp32" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\variants\twatch" "-IC:\Users\Mike\Documents\Arduino\libraries\TTGO_TWatch_Library-master\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\SPI\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\Ticker\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\Wire\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\FS\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\SPIFFS\src" "-IC:\Users\Mike\AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\BLE\src" "C:\Users\Mike\Documents\Arduino\libraries\TTGO_TWatch_Library-master\src\MPR121.cpp" -o nul -DARDUINO_LIB_DISCOVERY_PHASE