Thanks for that, I intuitively figured out there was something weird with serial monitor. So thank you for confirming that.
I also have cleaned out all the Arduino folders on my Mac and reinstalled. And that seems to have fixed other issues. I suspect that there was a failed update to libraries or corruption because after I cleaned out all the library folders (which are hidden unfortunately and they shouldn't be because it makes clean up difficult) the clean install 2.3.1 feels better now.
I have moved through so many versions now that its hard to say. And I wasn't benchmarking each one. So its just perception.
The other issues I'm having are related to changes in ESP32 2.x to 3.x which has broken so many things. But that is a different topic. But I wonder if the slowness is from the ESP32 libraries in 3.x .
I just got back into Arduino compiling for an ESP32S3, specifically a Heltec LoRa 32 v3.
Compiling the Example factory test sketches takes minutes, which feels like a lifetime.
No this has nothing to do with the spec of my laptop. About 6 years back I was using a an old Windows-8 laptop compiling a very long and complex mobile LTE firmware for an Express32, and it would complete in ~20 seconds. Would be great if you guys could come up with some more useful tips for reducing this compile waste of time. One thing I keep noticing is that many libraries keep compiling code that is never actually used withing the device. Another is that other code overwrites constants.
Perhaps you need to have a deeper look into how you optimize your libraries and the the very poor level of the example code. Often with no comments and no info of dependencies etc.
Yeah sure lot's of work, and nobody gets paid... But poor maintenance and code quality is no longer a valid excuse for FOSS.
UPDATE
If orgot to mention, that I also noticed that even though I have a 20 core CPU, Arduino-only seem to run on 1. Have you hard-coded the the IDE to only use single thread and/or 1 core? I also have not seen any indication of using multi-threaded compiling , is that not possible here?
Hi @E3V3A. I'm going to ask you to provide the full verbose output from a compilation.
This procedure is not intended to solve the problem. The purpose is to gather more information.
Please do this:
Select File > Preferences... (or Arduino IDE > Settings... for macOS users) from the Arduino IDE menus.
The "Preferences" dialog will open.
Check the box next to "Show verbose output during: ☐ compile" in the "Preferences" dialog.
Click the "OK" button.
The "Preferences" dialog will close.
Select Sketch > Verify/Compile from the Arduino IDE menus.
Wait for the compilation to finish.
Right click on the black "Output" panel at the bottom of the Arduino IDE window.
A context menu will open.
Select Copy All from the menu.
Open a forum reply here by clicking the "Reply" button.
Click the <CODE/> icon on the post composer toolbar.
This will add the forum's code block markup (```) to your reply to make sure the error messages are correctly formatted.
Press the Ctrl+V keyboard shortcut (Command+V for macOS users).
This will paste the compilation output into the code block.
Move the cursor outside of the code block markup before you add any additional text to your reply.
Click the "Reply" button to post the output.
In case the output is longer than the forum software will allow to be added to a post, you can instead save it to a .txt file and then attach that file to a reply here.
Select the .txt file you saved from the "Open" dialog.
Click the "Open" button.
The dialog will close.
Click the "Reply" button to publish the post.
Alternatively, instead of using the "Upload" icon on the post composer toolbar as described in steps (5) - (7) above, you can simply drag and drop the .txt file onto the post composer field to attach it.
Well, their build system is trying to be very smart, and spend lots of time parsing source files, creating prototypes (often wrong, if function has certain attribute(()) ) the Arduino parser just generates non-compilable code. Unfortunately it can not be switched off (I mean that library detecting, parsing, prototypes generation, glueing sources together etc). All these steps are very-very questionable but what we can do? Their build system copies source folders, edits your sources (without telling it to you). I am getting mad once a week when IDE decides to make autoindent (still have no clue when and why this happens) but does it wrong, creating ladders of code which I then fix back. Together with not-working "go-to-definition", "search all references" etc it is barely usable for something bigger than hello world.
If developers can fix these "go-to-reference", "go-to-h. file" , case sensitive search, make "library detection" optional, etc etc then, may be..
Right now I use Visual Slick Edit (I use it for 20 years or so) and compil;e with arduino-cli. It is not super fast but at least I got normal source editor which parses all .h/.c/.cpp files and creates tag files for quick search, quick go to references etc. My SlickEdit has indexed my project, ESP-IDF (sources), Libraries and even linux kernel sources. So in 1 or 2 clicks I can navigate through all the code
PS: I was thinking about posting bug reports but then decided not to do so: to many things to fix. Let it evolve in its own way
Hopefully a verbose compile log will provide some examples to analyze.
Please elaborate.
Simultaneous compile jobs -- up to the number of core threads -- happen when compiling libraries and the core. Also if you have non-.ino source in your sketch. Those results should also be cached and reused. The .ino is always recompiled; any separate .ino are combined into one.
If you run the arduino-cli back-end directly, you can force a --clean build, and set --jobs 1 to run in series to see how much worse that is. I have patched cli to show some basic timings. Compare the times for a complex ESP32 sketch using a dozen libraries
This is on a low-end quad-core running Windows 11, where just having the browser open (on a real web page) and doing nothing else is using up 15-20% CPU. Disabling parallel jobs doubles the time spent actually compiling. An immediate recompile still takes a chunk of time "detecting libraries" -- that is currently still single-threaded, but there has been work to make it multi-threaded as well, which cuts the time similarly. Linking is always a speed bump preventing a fast build. (Performance on the same box is noticeably better overall with Linux.)
It's a dozen libraries. Taking everything between Detecting libraries and Generating function prototypes in the verbose compile log: xtensa-esp32s3-elf-g++ is executed 124 times
each suuuper looong command line comprises the same 240-ish command-line switches, mostly include directories for the ESP32 core SDK and tools
here's a one-liner that will chop up those lines if you want to examine your own compile log
sed -e 's/\( -[[:alpha:]]\)/\n\1/g' -e 's/ \(@\?E:\)/\n \1/gi' compile.log
(My stuff is on drive E:; I ran that through WSL)
it happens in two parts (with no extra progress message to separate them)
first the ResolveLibrary phase, using the .cpp in the sketch. Each resolved library adds its src as yet another -I include path
runs once first with the generated .ino.cpp, resulting in the first library
10 more times with each library found, leaving two unresolved libraries
14 times, for the 12 other .cpp in my sketch (under src); two of them twice, after resolving the two other libraries
second part: 99 total .cpp files in the src directories for each library
This is single-threaded, and yes, you can watch as it takes about a second for each one on this machine. I just switched to using a Dev Drive and now it takes 109 seconds.
A brief explanation of the library discovery process is included in a PR to make it multi-threaded. Running off that PR, it takes 57 seconds: about the half the time with four cores. Only the second part (with 99 files) seems faster.
The same box running Linux took 36 and 19 seconds respectively for library discovery.
everybody knows that there is a problem somewhere and many dumps of the compiling output have been posted. Do not waste your time again !
Also the serial monitor has nothing to do with the compiling time.
No practical solution seems to be available today ...
We should be able to separate building the dependencies from building (or rebuilding) the sketch. When there are no changes to the global settings, to the library references, or to the library versions, then the compilation should not repeat the "ResolveLibrary" phase. A trivial change in a .INO or .CPP file should take a very short time to compile and link.
This procedure is not intended to solve the problem. The purpose is to gather more information.
) on the post composer toolbar:
