Encountered lousy boo-boo in arduino IDE generated compilation commands

Community Suggestions for the Arduino Project
1

You can't include a contributed library in a .cpp file without including a/any contributed library in the main .ino/.pde file!

I made simple tests by saving Blink into a regular sketch (not example any more) and adding a test.cpp tab to the file. I then added #include <phi_interfaces.h> in test.cpp but not in Blink.ino so this happens:

C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp.o 
C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp.o 
test.cpp:1:28: error: phi_interfaces.h: No such file or directory

Of course, the IDE failed to have this option on test.cpp!

-IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces

I then proceeded to add #include <phi_interfaces.h> in Blink.ino and this happens:

C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega -IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp.o 
C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega -IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp.o

Notice the difference in the library locations included? If I didn't include that contributed library in Blink.ino the IDE fails to see the need for this contributed library location in any .cpp files. If I do include the contributed library in Blink.ino, the IDE slaps the contributed library location to all file being compiled. This is lousy at best.

Arduino team, read this and made the change!

2

I've fought this issue for quite some time. It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

My opinion and preference would be to modify the IDE so that
the users personal library directory as well as the Arduino distribution directory is
added to the include path.

That way any code could include headers from any library by simply referencing the library's directory.

Example:
To include the Wire.h header file instead of using
#include <Wire.h>
it would be:
#include <Wire/Wire.h>
For spi it would be:
#include <SPI/SPI.h>

etc...
This would solve the issue you are seeing and make it really easy for libraries to reference other classes in other libraries,
which today really doesn't work unless the user adds the include files to his sketch,
which makes no sense since the sketch doesn't need it
but it does have the side effect of adding the specific library's directory to the include path.

If the existing goofy include path stuff was still done, but these two additional directories were added,
then naive users could continue to do their header includes the way they are now,
and those developing libraries or additional modules could use the library directory references to include
the needed headers without the users sketch having to include the headers.

It would be a very small backward compatible transparent change to the IDE that would add new
functionality going forward for those developing more complex s/w like libraries that depend
on or use other libraries.

--- bill

3

This mildly frustrates me too, the reason you have to do this is because there is no option to add additional include directories ( As you have found ).

I have other needs when it comes to changing the command line call to gcc.
I have been thinking of writing a module for my server app that will intercept all the IDE's interaction with the file system, then modify the avr-gcc.exe call with my new parameters. If I continue the project I'll forward you the finished app.

4

You could try using the mpide GUI to build the AVR sketches rather than the Arduino IDE.
mpide has additional capabilities in the boards.txt file
and introduces a platforms.txt which allows you to set custom defines and compiler
options on a per platform (AVR, pic32 etc..) or per board level within the platform.

There are also some decent makefiles out there for Arduino these days so you can bypass the IDE
completely.

--- bill

5

You could try using the mpide GUI to build the AVR sketches rather than the Arduino IDE.

I might give this a try, I would like to test some features of c++11 on the Arduino ( variadic templates, would revolutionize my library ).

On a side note, anyone read Italian and see the thread about upgrading the AVR system the IDE uses.
http://arduino.cc/forum/index.php/topic,96976.0.html

I might plug through it one day with Google translator.

6

Bill,

Yeah! It is OK for casual programmers especially those that don't know the difference between pde/ino and cpp

I've seen one possible solution to use #include <..\Wire\Wire.h> so if arduino team makes this change to all contributed library "import library", this may solve all the problem, will it?

pYro_65,

That'll be great! If you can intercept the commands and reassemble stuff to them, let us know!

At the moment I am developing user interfaces with LCD and keypads. I don't actually need any arduino hardware and simulated all necessary hardware (LCD, RTC, etc.) on PC :slight_smile: I only occasionally need to teach arduino IDE how to compile by including libraries in the main sketch (the main sketch doesn't need all those libraries as Bill mentioned). All my work has been on Dev C++. I can say whether to look in subfolders for include files. I don't know if it is an IDE thing or there is a g++ option to look inside subfolders. Dev C++ has a horrible debugger. I can't change variable values although it says I can. Addresses are often misinterpreted so I can't see what my strings look like. Plus the variable view doesn't update itself. I'm ready to move on to a different IDE. Man. Arduino is good but maybe it's my time to move my largish project to some better IDE. Any suggestions for a good PC-based IDE that uses gcc (for my simulator) and a good arduino-compatible avr-gcc IDE?

7

liudr:
I've seen one possible solution to use #include <..\Wire\Wire.h> so if arduino team makes this change to all contributed library "import library", this may solve all the problem, will it?

I tried using the relative path technique before on libraries.
It's been a while, and I recall having some kind of issue.
Like maybe it never built and pulled in the other library. (I can't remember)

Make sure to stay away from backwards slashes. Regular slashes is what should be used.
Normal slashes will always work in all OS environments - It is the C standard.
Backward slashes in file names/paths will only work in windows environments
that have special code to detect it.

--- bill

8

liudr:
Arduino is good but maybe it's my time to move my largish project to some better IDE.

I have had a good experience with the Arduino IDE, my project is currently 14 files long and 165kb in size and growing, the IDE has been able to handle it all. With access to the gcc command line I will have all I need. I have been meaning to contact you about your library in regards to a common serial interface for HW and SW systems ( you had a thread a while back I think ).

9

It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

10

PaulS:

It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

It is not a nightmare. Finding and resolving references is what the linker does.

Libraries are supposed to be independent entities whose modules can be compiled and archived into the library
archive independently.
By creating a system that requires an application to set up things a certain way in its source code
in order to be able to compile a library module the independence is broken.
This defeats the entire point of having a library. It is no longer really a library at that point.
And that is why having the include path used for compiling a library module be determined
by the sketch/application is a totally brain dead thing to do.

The way this should work (and does in other environments) is that library modules are compiled
and archived into a library archive independently from application code.
An application includes only the headers that it needs.
The app is compiled and then linked against library archives and any library functions needed
are pulled in from the archive by the linker.
If a library function called by the application needs another library function,
then those additional functions are also pulled in by the linker.
The application does not and should not have to be aware of the additional library dependencies
of the library functions the application uses.
Imagine how painful things would be if you had to do this include stuff to set up things in "normal" applications.
Can you imagine how many headers you would have to include in your windows app code and how long
it would take to build everything?

The arduino system has a system of extensible libraries. If the IDE and build process were done differently
then all the modules in each library would be built once (or whenever a library module was updated/added)
and archived into a library vs being built every single time the sketch is compiled.
Then a sketch would include only the headers it needs.
The sketch would be linked against the Arduino library archive and the needed references would be resolved.
It would not matter if the references were from the sketch or from a library module that the sketch pulled in.

The problem is that the IDE is simply not treating the libraries as a proper library.
If it did, then these kinds of interdependencies would go way and sketch building would
be much faster since only the modules that changed would need to be re-compiled and archived rather
than having to build every single module used by the sketch and its libraries.

--- bill

11

PaulS:

It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

Paul,

It's not just libraries. If I created a .cpp file and include a contributed library header, arduino IDE doesn't properly include that library's path when compiling the .cpp
There are times when I want to create classes in .h and .cpp but not necessarily make them arduino library. Say the classes are specific to only one project and other projects can't benefit from having the class in a library. Maybe I'm still working on the class and would keep it in a sketch folder so I can modify it in arduino IDE. There's more chances I can find to justify a better way of library links. I disagree with you on the fact the library should include all includes. That will be a very long list. I consider it better off if each library takes care of what it needs instead of what it's include needs.

12

It is not a nightmare. Finding and resolving references is what the linker does.

Finding and resolving references to installed libraries, yes. But the linker can't tell me what library I don't have installed.

If I copy a sketch posted on the forum/web/bathroom wall, the sketch currently must list all libraries needed. I like this feature. There is no need to install all the libraries listed in the sketch, only to find out that I need 14 more libraries, and then I need 7 more, and the 3 more, and then one more.

It's not just libraries. If I created a .cpp file and include a contributed library header, arduino IDE doesn't properly include that library's path when compiling the .cpp

The Arduino IDE makes a list of files to copy. That list includes all .ino/.pde files in the current directory, all files included by the sketches, and any .cpp files in the current directory that go with any header files in it's list. Then, it adds source files from libraries folders for header files that it does not yet have source files for.

When the list is complete, everything gets copied to one directory, and compiled there. There are no paths involved in the build process.

13

PaulS:
When the list is complete, everything gets copied to one directory, and compiled there. There are no paths involved in the build process.

Paul, yes I agree to this fact. I've seen the temp folder and know what are in it. What I was saying in my OP was that not every file copied to the same temp folder had the right -I option for g++ to even compile. The linker comes in after everything compiles but not everything is compiled when arduino IDE tells g++ to compile without proper header path.

14

PaulS:
There are no paths involved in the build process.

I think you need to have another look at how the build process works.
The library header files are included from their location in the "libraries" directories.
(user and system) by using an include path. This requires handing include paths to the compiler.

And that is one of the problems. The current build process requires that a separate include path
be given to the compiler for every single library used and those include paths are "learned" by
scanning through the users primary sketch file.
This is not a good way of doing things as it creates problems like what the Dr is seeing
and also keeps library modules from compiling that use other libraries unless the primary sketch
file has the additional, and unneeded by the sketch, header includes added to its source in order
to trick the IDE into modifying the include path to allow all the other modules to be able to locate
their header files.

While I wouldn't do that way, it is ok to scan the users sketch file to learn of libraries to try to make things
simpler for the users sketch file. Where I have the problem is requiring the user to modify his sketch file in order to get
library or other modules to compile.

--- bill

15

From my observation this is what I learned:

  1. The sketch files in the sketch folder are compiled with path to their required headers in alphabetic order.
  2. Then standard and contributed library .cpps are built in the order their headers show up in the main sketch. This includes things like LiquidCrystal or my own library.
  3. Then the arduino core library .cpps are build in an unknown fashion, probably in the order they were enumerated in a file or mentioned in arduino.h. This includes all the wiring stuff, hardware serial, etc.

All above compiled .o files reside in the same temp folder as Paul mentioned, including the source file in step 1. After that some linker options (I can't understand fully) to mention all the .o file to be linked.

16

There is another important step.
The .o files from the core files are all put into a real library archive called core.a
(you can see the archive commands "avr-ar" in the build output)
The final link is done by combining the sketch .o files with all the library .o files and then
linking against the core.a library archive and the math library.

Note that the core files are put into a real library while the "library" files are not.
The Arduino IDE does not treat the libraries and their module files like real library archive files.

What would be better is if all the library .o modules were also put into a library archive and then the users
.o files linked against the "libraries" library as well as the core library and the math library.

In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

--- bill

17

Liudr and bill
Bill thanks for writing down the missing library part in Liudr's story. Liudr, thanks for writing the main part.
I can confirm this is exactly as the Arduino IDE is working. I know for sure as I reproduced this process in Eclipse with my eclipse plugin.

bperrybap:
Note that the core files are put into a real library while the "library" files are not.
The Arduino IDE does not treat the libraries and their module files like real library archive files.

What would be better is if all the library .o modules were also put into a library archive and then the users
.o files linked against the "libraries" library as well as the core library and the math library.

I do not agree that it is better to treat all libraries as real libraries. The main reason is the extensive usage of defines. As the libraries in many cases can/will change (for instance because you select another arduino board.) rebuilding is the way to go.
Compiler optimizations that can be accomplished by using real libraries will be minimal as libraries in arduino are typically very dedicated. In other words: "If you use the library you need all of it".
This complexity has to be added for the arduino library or your project will literally explode. (It is hard to get something on a Uno)
So there is little benefit in adding this complexity dynamically for other libraries.

bperrybap:
In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

All compilers have a folder where you have your output files. In my humble opinion there are 2 reasons why the Arduino IDE uses temporary folders.

  1. to be able to separate output by build. In other words it is to much work to write code to manage the compile output. which is typically done by a compiler)
  2. Files can be build without saving. In other words you need 2 locations to save the file.

bperrybap:
The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

I agree on this. However I don't know how easy it would be to hide all the makefile complexity in the way the Arduino IDE does a great job now. I'm thinking about explaining "make clean" to a novice.

Best regards
Jantje

18

Jantje:
I do not agree that it is better to treat all libraries as real libraries. The main reason is the extensive usage of defines. As the libraries in many cases can/will change (for instance because you select another arduino board.) rebuilding is the way to go.

The biggest thing that treating all libraries as real libraries will do is force the IDE developers to look
at things just a little bit differently.

The way the IDE does its build now, you pay the price for separate module compilation,
without gaining the benefits like the ability to avoid re-compiling modules that have already been compiled.

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code form a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.

--- bill

With respect to rebuilding library modules:
To handle different "board" types, that is easily handled with sub directories
or different archives names in the build area based on board type.
If a board type is changed and the library archive for that board type does not exist it would be created.
But after that first time, it won't have to be built again.
Switch board types back to a board you were just using, and the library archive for that board type
will be up to date so no rebuilding is necessary.

When archive member (individual .o files) dependencies are in place,
if the archive member inside the archive does not exist or is out of date,
rules can indicate how to rebuild that archive member. (which will create the archive if it didn't exist)
Also, if any define that affects compilation is changed (which requires editing a file),
the corresponding modules will be re-compiled, then since the module's .o is newer than the one in the archive,
the .o in the archive will be updated.

Library archives are really nice, once everything is in place to use them.

Compiler optimizations that can be accomplished by using real libraries will be minimal as libraries in arduino are typically very dedicated. In other words: "If you use the library you need all of it".

I didn't follow the comment.
Using libraries, isn't about compiler optimizations, it is merely about organizing what is compiled
and how it is linked in.

This complexity has to be added for the arduino library or your project will literally explode. (It is hard to get something on a Uno)
So there is little benefit in adding this complexity dynamically for other libraries.

Huh? I didn't follow this.

Whether you link against libraries or against individual .o files, the final linked image does not change.
All the sames things can be done with .o file or library archives.
Libraries are simply a convenient and compact way to organize and store .o files.
A library is nothing more than an archive file of .o files.

bperrybap:
In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

All compilers have a folder where you have your output files. In my humble opinion there are 2 reasons why the Arduino IDE uses temporary folders.

  1. to be able to separate output by build. In other words it is to much work to write code to manage the compile output. which is typically done by a compiler)
  2. Files can be build without saving. In other words you need 2 locations to save the file.
    [/quote]

I was not arguing against using a temporary build area and sub directories.
I think having a build area is the way to go.
I was saying the copying of the source files to this build area should not necessary.
It should be possible to compile the code with the source files in place
and redirect the output objects to the very same place they are currently are.
(avoid the copy of the source files)

bperrybap:
The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

I agree on this. However I don't know how easy it would be to hide all the makefile complexity in the way the Arduino IDE does a great job now. I'm thinking about explaining "make clean" to a novice.
[/quote]

There shouldn't be a need to explain things like "make clean" to a novice.
The end user using an IDE on top of make should not see anything different from what they see today.
Make is used under the hood for its rule based build capabilities.
All the user would see different is that sketches would build much faster since
only the modified files would be re-compiled instead of all the files.

This is what other systems like Atmel's Visual Studio do.

19

bperrybap:
The way the IDE does its build now, you pay the price for separate module compilation,
without gaining the benefits like the ability to avoid re-compiling modules that have already been compiled.

That really depends on your definition of the word "now" :slight_smile:

Yes, that's how Arduino 1.0 works.

But the code on github, soon to become Arduino 1.0.1, has a patch I wrote many months ago which avoids recompiling code. It works more or less like you described, except it doesn't retain the .o files or a .a library for each board setting. It simply uses the .o files left over in the temporary directory.

This feature has existed in Teensyduino for nearly 1 year. Very soon it will be in Arduino.

Placing all the .o files for each library into a .a archive probably doesn't provide much benefit. If the archives were retained in a non-temporary directory, it could speed up the first compile when you begin running Arduino, or the first compile after changing to a different board (that's been used previously). That might be nice, even though it's a much less common case, but accomplishing that would take a pretty substantial patch. I doubt anyone will actually work on such a patch, and even if they did, I believe David probably would be reluctant to accept it.

One other thing to consider is where Arduino can write files. Long ago, it would create an "applet" subdirectory within the sketch folder and put the compiled files there. But that behavior caused all sorts of problems when compiling the examples, because many schools would install Arduino on a read-only networked filesystem. To solve that, the temporary directory was used for examples, but applet was used for normal sketches. Eventually, the "applet" subdirectory was removed, so Arduino would only attempt to write to the temporary folder. Writing all generated files to the temporary directory greatly improved satisfaction with Arduino among security conscious users, and the linux distros who now package Arduino to work across several "standard" locations.

Any proposal to store compiled archives must carefully consider the restrictive shared network filesystems which are used by many academic institutions. A patch which returns Arduino to the "bad old days" of writing within its own directories is certain to be rejected.

I was saying the copying of the source files to this build area should not necessary.
It should be possible to compile the code with the source files in place
and redirect the output objects to the very same place they are currently are.
(avoid the copy of the source files)

The library and core source files are never copied to the tempoary build directory. Only the sketch source is copied, and for good reason...

Arduino concatenates all the .ino & .pde files (if there's more than one) and does some preprocessing, mainly to automatically add #include <Aduino.h> and function declarations, which allows functions to call each other regardless of their order within the files.

Perhaps that preprocessing step could be done entirely within memory, with the result fed into the compiler via a pipe? But that would require quite a bit of extra java code. It's currently written in a fairly simple way, where all the .ino / .pde files are turned into a single (temporary) .cpp file, before the "Compile" step is even begun. While compiling, gcc is run using pretty much the same code with works on a paradigm of files being compiled to produce other files. If the preprocessed output were never written to disk, not only would the preprocessing code need to be changed to more complex code to manage the whole thing in memory, but the compile code would need to run gcc differently for one part where the input isn't a file, and the entire structure within Arduino would need change to pass the in-memory copy between the "Sketch" and "Editor" sections and the "Compile" object.

That would be a monstrous patch, only to avoid writing one temporary .cpp file!

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code form a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.

I have often considered writing a patch to fix this. It doesn't look like anyone else is ever going to do it....

I'd probably just reuse the existing parser that preprocesses the .pde / .ino files to extract a list of the #includes from all .cpp files, as if they'd been in the user's sketch. But if there's something special it should do, please enlighten me?

20

Yep, I've been using it for quite some time.(I liked the promotional video).
I'm very happy to see the Arduino team finally accepted it for the standard release.

Any proposal to store compiled archives must carefully consider the restrictive shared network filesystems which are used by many academic institutions. A patch which returns Arduino to the "bad old days" of writing within its own directories is certain to be rejected.

Compiled archives could live in the temporary build area.
This would essentially be a further optimization of of what Teensyduino was doing.
It would create an library archive for each board type as it is used and then
when possible re-use it when switching between board types or sketches.
Every time a new IDE session is started, since it gets a new build area, then the process starts clean.
But it would allow the library archive to expand during the IDE session.
For example, since the build directory does not change when you change sketches, then the core.a
and the library{boardtype}.a files would still exist and any libraries already compiled for that board
(even for a previous sketch during that IDE session) would not need to be compiled again.

Or even simpler, the IDE could build all the libraries for a board into an archive
inside the build area the fist time a sketch was was built.
It would be similar to the way the core library is built, only smarter.
Then, no library files would need to be rebuilt during that IDE session for that board.
In other words pay the library build time only on the first build (which would be a bit longer since it would build all libraries)
but then for the life of that IDE session builds are much faster even when you change sketches.

The library and core source files are never copied to the temporary build directory. Only the sketch source is copied, and for good reason...

I stand corrected on this. (Not sure how I overlooked that)
I had thought all the full tree for each library library was copied, but it does not appear to copy the sources.
It does seem to create a "utility" directory in each library directory even when it is not used for the library.
Does the IDE currently use the .d files it creates?
(I assume it must use them with the Teensyduino "smart build" update?)

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code from a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.

I have often considered writing a patch to fix this. It doesn't look like anyone else is ever going to do it....

I'd probably just reuse the existing parser that preprocesses the .pde / .ino files to extract a list of the #includes from all .cpp files, as if they'd been in the user's sketch. But if there's something special it should do, please enlighten me?

That really worries me, because I've run into several cases where perfectly valid C/C++ code in a sketch
will not get past the IDE parser. I've had to alter the sketch code to get it to build.
There are also cases where the IDE appears to have some magic code to allow setting some defines inside the sketch.
It appears that because of this it looks for things and where to insert the includes.
In many of my sketches, it often screws up where to place the includes especially if there are ifdefs.
There are also simple things like a missing double quote on a string that will cause the IDE parser to crash
and you get no error message like you would with the real compiler making the error difficult to locate.
I'd be ok with sharing the parser with the library code but I think the mainline parser needs some
updates before I'd feel comfortable running all the library modules through it.

The "something special" might be that if a library referenced another library then that referenced library
would need to be added to the list of libraries. But maybe that falls out when the same parser is used.

--- bill