Excessive time for simple build

Development Tools IDE 2.x
141

You can submit pull requests to propose improvements to the efficiency of the compilation process to the GitHub repository of the Arduino CLI code base:

https://arduino.github.io/arduino-cli/dev/CONTRIBUTING/#pull-requests

As for your idea of turning off the function prototype generation, I don't think such a proposal would be accepted. Like it or not, prototype generation is a fundamental part of the Arduino sketch framework. However, high quality proposals for improvements to the efficiency of this system that don't otherwise change the user experience or make the user interface more complex would surely be welcome.

142

Not every one appreciates the IDE leaving its junk behind after it exits.
It would be nice to have an IDE preference to control this behavior.

143

If for some reason the default behavior of potentially valuable caches being deleted after not being used for 30 days is a problem for you, you can configure the cache purging behavior by editing the configuration file @bperrybap. It is at this path:

~/.arduinoIDE/arduino-cli.yaml

Documentation for the configuration file is here:

https://arduino.github.io/arduino-cli/latest/configuration/#configuration-keys

144

One thing I have noticed over the past few years is that there is something broken with the ESP cores, particularly the ESP32.
It seems to re-compile things that it should not need to.
I have no idea how their board package works, but it may be blindly rebuilding some things in a script rather than being smarter about it.
It may be that they can't get the IDE to handle it using the IDE smarts so they just blindly do their rebuilds including recompiling files that should not need to be re-built.

On the size thing, that isn't going to change the ESP platforms have a much more sophisticated environment than other platforms so it is much larger.
That size should not make the actual builds slower after the initial build since it should only be recompiling the files that were modified since the previous build in the current IDE session.
But I like you I have seen the ESP32 core rebuild things that it should not need to be re-compiled.

145

The options only seem to affect how the cache is treated while the IDE is up and running.

There doesn't seem to be an option of fully clearing the cache when the IDE fully exits.
THAT is what I'm looking for. i.e. I don't want it to leave its turds on the disk after exiting.

146

Hi Mike,

I've started with Arduino few weeks ago and I am also a bit perplexed why IDE build system is so slow.

When I want to compile a single file it takes IDE 20 seconds just to tell me there is an error in my file. It checks all other project files, telling me "Using previously compiled file: ..." and so on.

At the moment I am looking into how to build binaries(.a) from all libraries I include (RTC, C++STL, LCD). This would be I think a way to go.
Ideally there will be an option in IDE to use .a instead of compiling all library code. For ArduinoSTL that is 40+ files.

E.g. there is core.a included so now I am trying to understand how to tell IDE to link my ArduinoSTL.a.

You can build library with command:

avr-ar rcs --plugin liblto_plugin-0.dll object_file_1.o object_file_2.o object_file_3.o ......

Frank

147

Hi @frank_11.

You can learn about how to use precompiled binaries in libraries from the Arduino Library Specification:

https://arduino.github.io/arduino-cli/latest/library-specification/#precompiled-binaries

1 Like
148

You care, but not enough to take the arduino IDEs sourcecode, make the necessary changes and submit a pull request. Nobody else but you will deem it necessary, so nobody will do it for you. If you have the budget you can pay somebody to do it, therwise it will not happen.

149

Hi @ptillisch,

Thank you for the link. I've seen that information - good to have a confirmation from you where to start with the next step. But I first needed to undestand how to build a library. Librarian was complaning about not having a plugin to cope with LTO object files so it took me some time to please it with adding '--plugin liblto_plugin-0.dll' - that I could not find anywhere.

150

You will have to polish your search skills :smiley: For an older version of ESP32; last line in below. As a Windows user you can use a tool like e.g. grepwin.

wim@AmiloPi1505:~/Downloads/arduino-1.8.19-testESP32-2.0.6/portable/packages/esp32/hardware/esp32/2.0.6/cores
$ grep -Rin '.' -e 'void setup'
./esp32/HardwareSerial.h:27:   void setup() {
./esp32/Arduino.h:137:void setup(void);

It's not necessarily in Arduino.h but in above it is.

You are an experienced programmer; do not compare that with an artist who knows nothing about programming but wants something easy or an absolute beginner.

Note:
Personally I think that it creates bad programming habits by taking that responsibility away from users. But it was a decision that was taken.

151

To be clear, the problem is not in the IDE front-end, but in the arduino-cli back-end, which is written in Go. It calls the the compilers like g++. Taking a quick look, the part in question looks innocuous enough

	b.logIfVerbose(false, tr("Detecting libraries used..."))
	err := b.libsDetector.FindIncludes(
		b.buildPath,
		b.buildProperties.GetPath("build.core.path"),
		b.buildProperties.GetPath("build.variant.path"),
		b.sketchBuildPath,
		b.sketch,
		b.librariesBuildPath,
		b.buildProperties,
		b.targetPlatform.Platform.Architecture,
	)
	if err != nil {
		return err
	}
	b.Progress.CompleteStep()

	b.warnAboutArchIncompatibleLibraries(b.libsDetector.ImportedLibraries())
	b.Progress.CompleteStep()

	b.logIfVerbose(false, tr("Generating function prototypes..."))
	if err := b.preprocessSketch(b.libsDetector.IncludeFolders()); err != nil {
		return err
	}
	b.Progress.CompleteStep()

(As of the last release; since then, the file has been moved.) For "Detecting libraries", at the latest commit, there are enticing bits like

	if !l.useCachedLibrariesResolution {
		sketch := sketch
		mergedfile, err := makeSourceFile(sketchBuildPath, sketchBuildPath, paths.New(sketch.MainFile.Base()+".cpp"))
		if err != nil {
			return err
		}
		sourceFileQueue.push(mergedfile)

		l.queueSourceFilesFromFolder(sourceFileQueue, sketchBuildPath, false /* recurse */, sketchBuildPath, sketchBuildPath)
		srcSubfolderPath := sketchBuildPath.Join("src")
		if srcSubfolderPath.IsDir() {
			l.queueSourceFilesFromFolder(sourceFileQueue, srcSubfolderPath, true /* recurse */, sketchBuildPath, sketchBuildPath)
		}

		for !sourceFileQueue.empty() {
			err := l.findIncludesUntilDone(cache, sourceFileQueue, buildProperties, sketchBuildPath, librariesBuildPath, platformArch)
			if err != nil {
				cachePath.Remove()
				return err
			}
		}

Seems like useCachedLibrariesResolution is always false -- if so, why is that?

"Generating function prototypes" is not about the .ino.cpp, where the magic has been applied. The verbose compiler output shows that file is already present during "Detecting libraries". Instead, it appears to create a very temporary sketch_merged.cpp, and then run ctags on it. This might be to enable the code hint/insight/sense/completion feature of the IDE, and therefore not strictly required to create the binary. If so, a better name for this step -- both the function name in the code and the message it prints -- might help; as would running it after the build and/or during the upload.

1 Like
152

Ctags is used to give Arduino CLI the understanding of the .ino file code it needs to perform the "sketch preprocessing" phase of the compilation (primarily the function prototype generation, but also inserting #line directives so that the compiler messages will match the filenames and line numbers in the sketch code instead of the C++ code that is actually being compiled).

Since the "code aware" features of Arduino IDE 2.x you mention are provided by a C++ language server, the sketch preprocessing that converts the "Arduino language" .ino file into valid C++ is an essential component of these features, but it is the clangd language server that actually provides the code aware features rather than Ctags.

Arduino IDE uses the "Arduino Language Server" tool to coordinate Arduino CLI and clangd.

As I mention above, it is required in order to convert the code in the .ino file into valid C++ that can be compiled by GCC.

153

So if you have a valid C++ sketch and don't need the code aware features, then I guess you could skip the sketch processing phase.

154

Possible misunderstanding here: I was not proposing that "function prototype generation" be turned off by default .. like you, I can see that is useful for starter-programmers.

I am just suggesting that it be a setting/option that can be selected by users of the IDE that have used C before [in my case since 1978]. This one setting would halve the 'compile' time for the 'blink' sketch (and presumably would be even more advantageous for larger sketches).

155

Definitely thanks for this! ESP is no longer a direction for me (this thread is a merge of ESP and Arduino-IDE topics). Completely agree about people who know nothing about programming .. that is exactly why I created Rexx :-). (See: https://speleotrove.com/rexxhist/TRL-background.pdf)

[sterretje edit]
fixed URL
[/sterretje edit]

156

I understand that and I'll repeat what I said before: it will not be accepted.

157

Hmmm. Halving the compile time would improve the user experience, I would have thought.

But thanks for warning me of that likely reaction.

1 Like
158

When I look at clangd.exe options, I see '-j 1'.
"c:\Program Files\Arduino IDE\resources\app\lib\backend\resources\clangd.exe" -log=verbose --pch-storage=memory -j 1

My Task manager shows that IDE does not use multiple threads during the build process. That might speed up the dependency search/code copilation.

task manager arduino

159

Sorry, my bad. I see clangd is not responsible for build process. It is arduino-cli.exe right ? It has a config file .arduinoIDE\arduino-cli.yaml. Is there an option to tell it to use multiple threads ?

160

Well, I got stuck in to developing my project, using the Arduino IDE. As soon as I added a new device (the SparkFun-arduino-MLX90393) the verify time went back to over a minute.

I tried using this for a couple days .. but it became impossible. Changing a few lines of code and then verifying is not a viable way of working with the IDE as it stands: the verify time is ten or more times longer than the editing time. This might be acceptable for a first-ever getting-to-know-the-IDE 'toy' project, but the IDE is not fit for purpose for serious programming.

Surely there must be some way to get a reasonable verify performance (a few seconds at most) using the IDE?

1 Like