Why always recompiling unchanged library code

Development Tools IDE 2.x
1

Hi,

When I look at the verbose information during compilation of my source-code, I noticed that installed libs are always recompiled, in a few cases a cached version is used. The compilation of one small change in my own code will take up to minutes before it can be uploaded to my board. And yes, I used the latest processor in my Mac development computer (M4). So computer power is not the issue.

With other compilers (e.g., GNU CC) I'm used to generate lib-files of stable code, such as the the installed Arduino/ESP32 libs. How can I establish this in the Arduino environment and thereby decreasing significantly the compilation time ?

Best regards, Ronald.

2

At the moment you can't. I think it's a reported issue, check on github if you want to know then click on subscribe to get updated. IIRC, it's low priority so could be a while. If you want to really speed up your compiles turn off the Verify and Upload verbose output.
Screenshot 2025-05-29 at 09.22.02

1 Like
3

Can you give an example of a library plus how to install it (external site, library manager, ...) that always recompiles? Please also provide the board / board settings that you used.

4

As indicated, not a specific lib but observation in general. It seems the IDE checks all dependencies while as a programmer you know what changes and not. So libs would be a trivial option. I'm almost tending to use the CLI and investigate how I
can use gcc to generate libs. I lose so much time waiting. :grinning_face:

5

Thnx. Interesting and helpful response.

6

You should find that a second and more compilation with the output turned off goes really fast.

7

It also concerns espressif basic stuff to be honest. All custom libraries are all ready eliminated. Checking the forum I see many "worries" to say it mildly on this topic without a proper solution. So I assume these worries remain for a long time. Really cumbersome when you are calibrating some values in your code. Anyway, ...

8

I don't have ESP32 based boards but can compile for them.

Board: ESP Dev module
Sketch: bt_classic_device_discovery (Examples for ESP32 Dev Module).

First compile:

Compiling library "BluetoothSerial"
"C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp-x32\\2411/bin/xtensa-esp32-elf-g++" -MMD -c "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/cpp_flags" -Wall -Wextra -Os -Werror=return-type -DF_CPU=240000000L -DARDUINO=10607 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 "-DARDUINO_BOARD=\"ESP32_DEV\"" "-DARDUINO_VARIANT=\"esp32\"" -DARDUINO_PARTITION_default "-DARDUINO_HOST_OS=\"windows\"" "-DARDUINO_FQBN=\"esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=none,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default\"" -DESP32=ESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DARDUINO_USB_CDC_ON_BOOT=0 "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/defines" "-IC:\\Users\\bugge\\AppData\\Local\\Temp\\.arduinoIDE-unsaved2025429-9372-11r1tff.69y7\\bt_classic_device_discovery" -iprefix "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/include/" "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/includes" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/qio_qspi/include" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\cores\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\variants\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/build_opt.h" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/file_opts" "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src\\BluetoothSerial.cpp" -o "C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41\\libraries\\BluetoothSerial\\BluetoothSerial.cpp.o"
"C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp-x32\\2411/bin/xtensa-esp32-elf-g++" -MMD -c "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/cpp_flags" -Wall -Wextra -Os -Werror=return-type -DF_CPU=240000000L -DARDUINO=10607 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 "-DARDUINO_BOARD=\"ESP32_DEV\"" "-DARDUINO_VARIANT=\"esp32\"" -DARDUINO_PARTITION_default "-DARDUINO_HOST_OS=\"windows\"" "-DARDUINO_FQBN=\"esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=none,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default\"" -DESP32=ESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DARDUINO_USB_CDC_ON_BOOT=0 "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/defines" "-IC:\\Users\\bugge\\AppData\\Local\\Temp\\.arduinoIDE-unsaved2025429-9372-11r1tff.69y7\\bt_classic_device_discovery" -iprefix "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/include/" "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/includes" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/qio_qspi/include" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\cores\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\variants\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/build_opt.h" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/file_opts" "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src\\BTAddress.cpp" -o "C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41\\libraries\\BluetoothSerial\\BTAddress.cpp.o"
"C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp-x32\\2411/bin/xtensa-esp32-elf-g++" -MMD -c "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/cpp_flags" -Wall -Wextra -Os -Werror=return-type -DF_CPU=240000000L -DARDUINO=10607 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 "-DARDUINO_BOARD=\"ESP32_DEV\"" "-DARDUINO_VARIANT=\"esp32\"" -DARDUINO_PARTITION_default "-DARDUINO_HOST_OS=\"windows\"" "-DARDUINO_FQBN=\"esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=none,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default\"" -DESP32=ESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DARDUINO_USB_CDC_ON_BOOT=0 "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/defines" "-IC:\\Users\\bugge\\AppData\\Local\\Temp\\.arduinoIDE-unsaved2025429-9372-11r1tff.69y7\\bt_classic_device_discovery" -iprefix "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/include/" "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/includes" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/qio_qspi/include" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\cores\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\variants\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/build_opt.h" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/file_opts" "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src\\BTAdvertisedDeviceSet.cpp" -o "C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41\\libraries\\BluetoothSerial\\BTAdvertisedDeviceSet.cpp.o"
"C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp-x32\\2411/bin/xtensa-esp32-elf-g++" -MMD -c "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/cpp_flags" -Wall -Wextra -Os -Werror=return-type -DF_CPU=240000000L -DARDUINO=10607 -DARDUINO_ESP32_DEV -DARDUINO_ARCH_ESP32 "-DARDUINO_BOARD=\"ESP32_DEV\"" "-DARDUINO_VARIANT=\"esp32\"" -DARDUINO_PARTITION_default "-DARDUINO_HOST_OS=\"windows\"" "-DARDUINO_FQBN=\"esp32:esp32:esp32:UploadSpeed=921600,CPUFreq=240,FlashFreq=80,FlashMode=qio,FlashSize=4M,PartitionScheme=default,DebugLevel=none,PSRAM=disabled,LoopCore=1,EventsCore=1,EraseFlash=none,JTAGAdapter=default,ZigbeeMode=default\"" -DESP32=ESP32 -DCORE_DEBUG_LEVEL=0 -DARDUINO_RUNNING_CORE=1 -DARDUINO_EVENT_RUNNING_CORE=1 -DARDUINO_USB_CDC_ON_BOOT=0 "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/defines" "-IC:\\Users\\bugge\\AppData\\Local\\Temp\\.arduinoIDE-unsaved2025429-9372-11r1tff.69y7\\bt_classic_device_discovery" -iprefix "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/include/" "@C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/flags/includes" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\tools\\esp32-arduino-libs\\idf-release_v5.4-2f7dcd86-v1\\esp32/qio_qspi/include" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\cores\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\variants\\esp32" "-IC:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/build_opt.h" "@C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41/file_opts" "C:\\Users\\bugge\\AppData\\Local\\Arduino15\\packages\\esp32\\hardware\\esp32\\3.2.0\\libraries\\BluetoothSerial\\src\\BTScanResultsSet.cpp" -o "C:\\Users\\bugge\\AppData\\Local\\arduino\\sketches\\616F70C8A1189478903E7F15414F4D41\\libraries\\BluetoothSerial\\BTScanResultsSet.cpp.o"

Second compile

Compiling libraries...
Compiling library "BluetoothSerial"
Using previously compiled file: C:\Users\bugge\AppData\Local\arduino\sketches\616F70C8A1189478903E7F15414F4D41\libraries\BluetoothSerial\BTAddress.cpp.o
Using previously compiled file: C:\Users\bugge\AppData\Local\arduino\sketches\616F70C8A1189478903E7F15414F4D41\libraries\BluetoothSerial\BluetoothSerial.cpp.o
Using previously compiled file: C:\Users\bugge\AppData\Local\arduino\sketches\616F70C8A1189478903E7F15414F4D41\libraries\BluetoothSerial\BTAdvertisedDeviceSet.cpp.o
Using previously compiled file: C:\Users\bugge\AppData\Local\arduino\sketches\616F70C8A1189478903E7F15414F4D41\libraries\BluetoothSerial\BTScanResultsSet.cpp.o

This shows that libraries are not recompiled if they were not changed.

Attached the full compile logs

firstCompile.txt (169.3 KB)
secondCompile.txt (19.9 KB)

9

This does not answer the question. The question is: why do you check all dependencies when you already know in advance that a library hasn't changed?

The question went a bit deeper than simply proving that compilation works, whether or not cached code is used. That was known. The question is deeper in the sense that it involves understanding how the Arduino IDE handles code, something you could read and grasp in several places. The Arduino IDE first concatenates all (user) code into a temporary file, then generates all forward declarations, and checks all dependencies — even those that haven’t changed, which as a skilled programmer you can already know. It’s a bit convoluted and unnecessary. And thereby time consuming (even when using cached code).

The reason is clear (as described in many sources): the Arduino IDE is designed for hobbyists, students, and teachers. Those who don’t count themselves among that group (i.e., skilled or experienced programmers) tend to choose a different IDE.

I’ve now done an experiment with PlatformIO. What a relief: the same code takes 2 minutes to build in the Arduino IDE, but only 3.05 seconds in PlatformIO. PlatformIO also respects proper C++ code, whereas the Arduino IDE is overly lenient with .ino files.... so to say.

Anyway, thanks @sonofcy for understanding. Maybe PlatformIO is also recommendable for you.

10

I would like to use PlatformIO and have tried but I always seem to mess up the install. Keep in mind 83yo with ASD. I may be misunderstanding something, you say

How do 'they' know in advance?

One thing I think I have noticed is that AVR code compiles the fastest and (going from old memory cells) seems to recognize unchanged libs.

One caution: if you think these are single-user computers, that is not entirely true. Remote access from a second user can occur; I have had as many as three additional users. Even a single person can be running two copies of the IDE. I am reasonably confident my observations are correct, and the developers, knowing that, may be facing more difficult rules to follow than we realize. However, I join you in urging them to resolve this situation as soon as possible.

One question I have is, do we know if a file is compiled if the date check indicates it is unchanged? It just occurred to me that the problem may be that the compile output isn't retained, so a check is sometimes impossible.
I have a suggestion: why not give users a configuration option for how much disk space to allocate for permanent compiler output, along with a way to clean it up based on date, size, or other criteria?

11

BTW: your response was oke.

When you install GNUC you install obviously libraries that you may use. They don' change. When you use a library fir arduino they normally do nit change. But since libraries for arduinio come in source code you have to compile them once. So after that you can assume they are correct until you modify them.

12

Unless they get auto-updated. There is also the issue of disk space. The Arduino Corp does not want to use all of your disk space to store the library object code. I think that is the key reason you are experiencing the recompile phenomenon.

13

Indeed, I only reacted on the beliw statement from the opening post

14

I am guessing you are referring to library discovery, where Arduino IDE automatically determines the library dependencies of the sketch:

https://arduino.github.io/arduino-cli/latest/sketch-build-process/#dependency-resolution

I think you have an overly simple conception of what "haven't changed" means. The entire set of installed libraries are a factor in library discovery, so if the user has installed any new libraries since the previous compilation, this could be significant even if the specific library that happened to be used during the previous compilation didn't change.

I guess this could be handled by storing the modification timestamp of each of the libraries installation folders in the sketch build folder, then on subsequent compilations of the sketch, checking whether any of the libraries folders have a different timestamp. If not, the cached library discovery results from the previous compilation can be reused instead of running the discovery process over again.

15

I think it is a safe assumption. The actual system in use is somewhat similar, except it uses the less reliable approach of only checking whether the cached object's modification timestamp is newer than the object's source file dependencies:

However, this system can only be used after the dependencies of the sketch are known, so the library discovery data for the sketch must be obtained before the system can start to check whether it can use cached objects for those dependencies instead of compiling them from source.

16

No, I do not have an overly simplified conception of. Maybe the title was somewhat pronounced... but I think that the/ my issue was clarified in sufficient detail. When a compilation time difference between Arduino IDE and (for example) PlatformIO, resp 2 minutes versus 3 -5 seconds... then the waiting time is a waste of time. Also the poor adherence to the ISO CPP is a serious shortcoming. But, anyone can choose his/her own toolset.

For me this thread can be closed.