Call for beta testers: Arduino CLI sketch compilation optimization

Software Arduino command line tools
1

TL;DR

In order to help understand the context and importance of this beta testing initiative, I will provide some background information below. If you aren't interested in the boring details at this time and just want to get to the beta testing, feel free to skip over that "Background" section and go straight to the "Call for Beta Testers!" section below.

Background

Compilation Optimization

A common complaint from the Arduino community is of unexpectedly long durations of sketch compilation/builds.

The developers have done some significant work to improve on the performance in the past. This work included parallelization of the compilation processes, and a cache system, where the objects produce by a compilation are stored, then reused in subsequent compilations when appropriate. This means that the actual compilation phase of the sketch build is fairly optimized at this point and there isn't a lot of low hanging fruit left to improve there.

Library Discovery

There is another significant phase of the sketch build process, which is the library discovery: In order to make Arduino project development accessible to everyone, the sketch build system automatically configures the compiler's "search path" with the paths of each of the libraruy dependencies of the sketch. This is a significant difference from traditional C++ development, where the user must manually configure these paths (which would be incredibly difficult for the average Arduino beginner to accomplish).

Library discovery is accomplished done by repeatedly preprocessing the sketch and then searching the library installation folders for the library that provides the header files that were not found by the compiler, adding that path to the compiler's search path, the repeating until the preprocessing is successful (see the flowchart here for an overview of the process). This fairly resource intensive process is performed on every compilation.

This phase produces the content you will see under the "Detecting libraries used..." section of the verbose compilation. For example:

Detecting libraries used...
C:\Users\per\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega32u4 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_LEONARDO -DARDUINO_ARCH_AVR -DUSB_VID=0x2341 -DUSB_PID=0x8036 -DUSB_MANUFACTURER="Unknown" -DUSB_PRODUCT="Arduino Leonardo" -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\cores\arduino -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\variants\leonardo C:\Users\per\AppData\Local\Temp\arduino\sketches\C2DF9AB8624136325E85A8703D7AF1C8\sketch\ManyIncludes.ino.cpp -o nul
Alternatives for Ethernet.h: [Ethernet@2.0.2]
ResolveLibrary(Ethernet.h)
  -> candidates: [Ethernet@2.0.2]
C:\Users\per\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega32u4 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_LEONARDO -DARDUINO_ARCH_AVR -DUSB_VID=0x2341 -DUSB_PID=0x8036 -DUSB_MANUFACTURER="Unknown" -DUSB_PRODUCT="Arduino Leonardo" -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\cores\arduino -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\variants\leonardo -IC:\Users\per\Documents\Arduino\libraries\libraries\Ethernet\src C:\Users\per\AppData\Local\Temp\arduino\sketches\C2DF9AB8624136325E85A8703D7AF1C8\sketch\ManyIncludes.ino.cpp -o nul
Alternatives for Keyboard.h: [Keyboard@1.0.6]
ResolveLibrary(Keyboard.h)
  -> candidates: [Keyboard@1.0.6]
C:\Users\per\AppData\Local\Arduino15\packages\arduino\tools\avr-gcc\7.3.0-atmel3.6.1-arduino7/bin/avr-g++ -c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -E -CC -mmcu=atmega32u4 -DF_CPU=16000000L -DARDUINO=10607 -DARDUINO_AVR_LEONARDO -DARDUINO_ARCH_AVR -DUSB_VID=0x2341 -DUSB_PID=0x8036 -DUSB_MANUFACTURER="Unknown" -DUSB_PRODUCT="Arduino Leonardo" -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\cores\arduino -IC:\Users\per\AppData\Local\Arduino15\packages\arduino\hardware\avr\1.8.6\variants\leonardo -IC:\Users\per\Documents\Arduino\libraries\libraries\Ethernet\src -IC:\Users\per\Documents\Arduino\libraries\libraries\Keyboard\src C:\Users\per\AppData\Local\Temp\arduino\sketches\C2DF9AB8624136325E85A8703D7AF1C8\sketch\ManyIncludes.ino.cpp -o nul

[...]

(note that a new -I flag for the previously discovered library is added to the avr-g++ -E command at each iteration)

Library Discovery Optimization Work

The Arduino CLI developers have been working on making the library discovery process multi-threaded in order to reduce the duration of the process:

This work will benefit users of Arduino IDE and Arduino Cloud Editor in addition to those who use Arduino CLI directly.

Call for Beta Testers!

Since this is a fairly complex change, we are hoping the community can help us to get good testing coverage across a variety of real world environments and sketches. I'll provide instructions you can follow to get the tester build that contains this optimization work:

  1. Sign in to your GitHub account.
    (GitHub only allows downloads of the tester builds when you are signed in.)
  2. Open this page in your browser:
    https://github.com/arduino/arduino-cli/actions/runs/10305203157?pr=2625#artifacts
  3. Click the appropriate link for your operating system under the "Artifacts" section of the page:
    • If you are using Linux: "Linux_X86-64"
    • If you are using macOS (Apple Silicon processor): "macOS_ARM64"
    • If you are using Windows: "Windows_X86-64"
  4. Wait for the download to finish.
  5. Extract the downloaded file.
  6. Extract the ZIP file you find inside the extracted file.
    (yes, there are two layers of ZIP files :upside_down_face:)

Now use the arduino-cli executable you find in the extracted folder to compile your sketches. Please let us know if you encounter any problems. You can report them by commenting on the pull request thread here:

https://github.com/arduino/arduino-cli/pull/2625

If you have any questions, concerns, or problems regarding installing or using this tester build, comment here on the forum thread and I'll be happy to provide assistance.

ⓘ The tester build of Arduino CLI won't interfere with any other installations of Arduino CLI or Arduino IDE on your computer so there is no problem using beta tester versions of Arduino CLI installed in addition to the release version you use normally.

2 Likes
2

I’m on the go at the moment so can’t test but do have a question

Couldn’t you cache this in some way ?

3

Installed the ARMv7 version and so far no problems on a variety of cores.

1 Like
4

The library detection is cached to some degree. I can't comment on correctness -- although it did compile -- but here are the times, before and after (single- and multi-threaded); on an ESP32 sketch I profiled previously; for a clean Verify and immediate re-Verify.

== Single-threaded     clean      cache
prepare                 0.850      0.985
detecting libraries    36.422      5.256
function prototypes     0.861      0.892
compiling sketch        5.771      1.437
compiling libraries    25.945      0.205
compiling core          5.550      0.212
linking                 5.371      5.168
final                   0.611      0.604
                     --------   --------
Total                  81.380     14.759

== Multi-threaded      clean      cache
prepare                 0.850      0.973
detecting libraries    19.225      2.928
function prototypes     0.860      0.856
compiling sketch        5.411      1.427
compiling libraries    25.125      0.194
compiling core          5.534      0.232
linking                 5.343      5.185
final                   0.622      0.576
                     --------   --------
Total                  62.970     12.370

Four cores, and the detection times are roughly half. About the same improvement as with multi-threaded compilation (which you can disable) that I had tested earlier.

2 Likes
5

Promising .. I haven't used CLI so cannot directly compare. Is there a way to persuade the IDE to use this?

Mike

6

Unfortunately it is not easily done.

The Arduino IDE application's code is written for use with Arduino CLI version 0.36.0.

Unrelated to the performance optimization proposed here, there have been some significant changes in the API Arduino IDE uses to communicate with Arduino CLI since the time of the Arduino CLI 0.36.0 release. The Arduino IDE application's code must be updated for the new API before it can be used with newer versions of Arduino CLI.

The developers are working on that:

However, I think that it is still a work in progress.

So it is probably most simple to just use Arduino CLI directly for beta testing this proposal. If you have any questions about using Arduino CLI, I'd be happy to answer them. There is a general overview here:

https://arduino.github.io/arduino-cli/latest/getting-started/

7

OK, thanks .. I think I did try it earlier this year but it wasn't faster that the IDE compile. Will take another look.

[Aside: 95+% of the time I'd just be doing a Verify, which should be easy and fast. So maybe all I really need is a way to stop the 'locating libraries' and use explicit #includes?]

8

When you compile a sketch in Arduino IDE, it just sends a command to Arduino CLI to compile the sketch, just the same as if you ran the equivalent arduino-cli compile command from a terminal.

The only difference is that when you are compiling with Arduino IDE, you have the system overhead of running the Arduino IDE application itself. On a system that doesn't have a lot of resources, there might not much left over for the compilation process after the overhead from running the Arduino IDE application. But on a system with a decent CPU and RAM, you probably wouldn't notice any difference if you compare compiling with Arduino IDE 2.3.2 against compiling with Arduino CLI 0.36.0.

But the situation might be different when comparing compiling with Arduino IDE 2.3.2 against compiling with the beta tester version of Arduino CLI that has the threaded library discovery, since in this case you are essentially comparing the performance of Arduino CLI 0.36.0 against the performance of the beta tester version of Arduino CLI.

9

Understood. That's why the beta CLI sounds interesting.

But I'd prefer to completely bypass the library discovery if possible.

10

Looks like this is now used in 2.3.3. The 'Detecting Libraries' is noticeably faster (3x or 4x on my 4-core PC) and the whole Verify (once indexing is over and the core compiled) is reliably under 40 seconds.

Progress! Many thanks to everyone involved in making this happen!

I'll pick up my project again soon.

Mike

11

I'm glad you have found the performance of 2.3.3 to be better!

The Arduino CLI developers definitely did some optimization work that was shipped in Arduino IDE 2.3.3 through the update of its Arduino CLI dependency from 0.35.3 to 1.0.4. However, the optimization discussed in this topic is not in Arduino IDE 2.3.3. So beta testing is still welcome. We didn't receive much testing on the proposed optimization and I think this has made the developers nervous to move forward with it. So the lack of testers is one of the reasons why it didn't end up going into Arduino IDE 2.3.3.

Although it is unfortunate that we missed it in Arduino IDE 2.3.3, the fact that it is still to come is actually good news in a way because it means there is potentially even more improvement beyond what you already saw coming in a future version of Arduino IDE.

1 Like