Go Down

Topic: Encountered lousy boo-boo in arduino IDE generated compilation commands (Read 6658 times) previous topic - next topic

bperrybap

There is another important step.
The .o files from the core files are all put into a *real* library archive called core.a
(you can see the archive commands "avr-ar" in the build output)
The final link is done by combining the sketch .o files with all the library .o files and then
linking against the core.a library archive and the math library.

Note that the core files are put into a real library while the "library" files are not.
The Arduino IDE does not treat the libraries and their module files like real library archive files.

What would be better is if all the library .o modules were also put into a library archive and then the users
.o files linked against the "libraries" library as well as the core library and the math library.

In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

--- bill

Jantje

Liudr and bill
Bill thanks for writing down the missing library part in Liudr's story. Liudr, thanks for writing the main part.
I can confirm this is exactly as the Arduino IDE is working. I know for sure as I reproduced this process in Eclipse with my eclipse plugin.



Note that the core files are put into a real library while the "library" files are not.
The Arduino IDE does not treat the libraries and their module files like real library archive files.

What would be better is if all the library .o modules were also put into a library archive and then the users
.o files linked against the "libraries" library as well as the core library and the math library.

I do not agree that it is better to treat all libraries as real libraries. The main reason is the extensive usage of defines. As the libraries in many cases can/will change (for instance because you select another arduino board.) rebuilding is the way to go.
Compiler optimizations that can be accomplished by using real libraries will be minimal as libraries in arduino are typically very dedicated. In other words: "If you use the library you need all of it".
This complexity has to be added for the arduino library or your project will literally explode. (It is hard to get something on a Uno)
So there is little benefit in adding this complexity dynamically for other libraries.


In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

All compilers have a folder where you have your output files. In my humble opinion there are 2 reasons why the Arduino IDE uses temporary folders.
1) to be able to separate output by build. In other words it is to much work to write code to manage the compile output. which is typically done by a compiler)
2) Files can be build without saving. In other words you need 2 locations to save the file.


The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

I agree on this.  However I don't know how easy it would be to hide all the makefile complexity in the way the Arduino IDE does a great job now. I'm thinking about explaining "make clean" to a novice.

Best regards
Jantje
Do not PM me a question unless you are prepared to pay for consultancy.
Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -

bperrybap


I do not agree that it is better to treat all libraries as real libraries. The main reason is the extensive usage of defines. As the libraries in many cases can/will change (for instance because you select another arduino board.) rebuilding is the way to go.


The biggest thing that treating all libraries as *real* libraries will do is force the IDE developers to look
at things just a little bit differently.

The way the IDE does its build now, you pay the price for separate module compilation,
without gaining the benefits like the ability to avoid re-compiling modules that have already been compiled.

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code form a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.



--- bill





With respect to rebuilding library modules:
To handle different "board" types, that is easily handled with sub directories
or different archives names in the build area based on board type.
If a board type is changed and the library archive for that board type does not exist it would be created.
But after that first time, it won't have to be built again.
Switch board types back to a board you were just using, and the library archive for that board type
will be up to date so no rebuilding is necessary.

When archive member (individual .o files) dependencies are in place,
if the archive member inside the archive does not exist or is out of date,
rules can indicate how to rebuild that archive member. (which will create the archive if it didn't exist)
Also, if any define that affects compilation is changed (which requires editing a file),
the corresponding modules will be re-compiled, then since the module's .o is newer than the one in the archive,
the .o in the archive will be updated.

Library archives are really nice, once everything is in place to use them.

Quote

Compiler optimizations that can be accomplished by using real libraries will be minimal as libraries in arduino are typically very dedicated. In other words: "If you use the library you need all of it".


I didn't follow the comment.
Using libraries, isn't about compiler optimizations, it is merely about organizing what is compiled
and how it is linked in.


Quote

This complexity has to be added for the arduino library or your project will literally explode. (It is hard to get something on a Uno)
So there is little benefit in adding this complexity dynamically for other libraries.


Huh? I didn't follow this.

Whether you link against libraries or against individual .o files, the final linked image does not change.
All the sames things can be done with .o file or library archives.
Libraries are simply a convenient and compact way to organize and store .o files.
A library is nothing more than an archive file of .o files.


In fact with proper compiler options, there is no need to do the silly file copying that the IDE is doing
for the library source files.

All compilers have a folder where you have your output files. In my humble opinion there are 2 reasons why the Arduino IDE uses temporary folders.
1) to be able to separate output by build. In other words it is to much work to write code to manage the compile output. which is typically done by a compiler)
2) Files can be build without saving. In other words you need 2 locations to save the file.
[/quote]

I was not arguing against using a temporary build area and sub directories.
I think having a build area is the way to go.
I was saying the copying of the source files to this build area should not necessary.
It should be possible to compile the code with the source files in place
and redirect the output objects to the very same place they are currently are.
(avoid the copy of the source files)


The IDE is great example of why make and makefiles are so useful and better at doing builds.
The ideal would have been to build the IDE on top of make so that the all the IDE has to do is
build a stub sketch makefile and then let the make and a primary makefile handle all the rules for the actual build.
This would have allowed making tweaks to the build process without having to modify the actual JAVA code.

I agree on this.  However I don't know how easy it would be to hide all the makefile complexity in the way the Arduino IDE does a great job now. I'm thinking about explaining "make clean" to a novice.
[/quote]

There shouldn't be a need to explain things like "make clean" to a novice.
The end user using an IDE on top of make should not see anything different from what they see today.
Make is used under the hood for its rule based build capabilities.
All the user would see different is that sketches would build much faster since
only the modified files would be re-compiled instead of all the files.

This is what other systems like Atmel's Visual Studio do.




pjrc

#18
Mar 28, 2012, 02:05 pm Last Edit: Mar 28, 2012, 03:40 pm by Paul Stoffregen Reason: 1

The way the IDE does its build now, you pay the price for separate module compilation,
without gaining the benefits like the ability to avoid re-compiling modules that have already been compiled.


That really depends on your definition of the word "now"  :)

Yes, that's how Arduino 1.0 works.

But the code on github, soon to become Arduino 1.0.1, has a patch I wrote many months ago which avoids recompiling code.  It works more or less like you described, except it doesn't retain the .o files or a .a library for each board setting.  It simply uses the .o files left over in the temporary directory.

This feature has existed in Teensyduino for nearly 1 year.  Very soon it will be in Arduino.

Placing all the .o files for each library into a .a archive probably doesn't provide much benefit.  If the archives were retained in a non-temporary directory, it could speed up the first compile when you begin running Arduino, or the first compile after changing to a different board (that's been used previously).  That might be nice, even though it's a much less common case, but accomplishing that would take a pretty substantial patch.  I doubt anyone will actually work on such a patch, and even if they did, I believe David probably would be reluctant to accept it.

One other thing to consider is where Arduino can write files.  Long ago, it would create an "applet" subdirectory within the sketch folder and put the compiled files there.  But that behavior caused all sorts of problems when compiling the examples, because many schools would install Arduino on a read-only networked filesystem.  To solve that, the temporary directory was used for examples, but applet was used for normal sketches.  Eventually, the "applet" subdirectory was removed, so Arduino would only attempt to write to the temporary folder.  Writing all generated files to the temporary directory greatly improved satisfaction with Arduino among security conscious users, and the linux distros who now package Arduino to work across several "standard" locations.

Any proposal to store compiled archives must carefully consider the restrictive shared network filesystems which are used by many academic institutions.  A patch which returns Arduino to the "bad old days" of writing within its own directories is certain to be rejected.


Quote

I was saying the copying of the source files to this build area should not necessary.
It should be possible to compile the code with the source files in place
and redirect the output objects to the very same place they are currently are.
(avoid the copy of the source files)



The library and core source files are never copied to the tempoary build directory.  Only the sketch source is copied, and for good reason...

Arduino concatenates all the .ino & .pde files (if there's more than one) and does some preprocessing, mainly to automatically add #include <Aduino.h> and function declarations, which allows functions to call each other regardless of their order within the files.

Perhaps that preprocessing step could be done entirely within memory, with the result fed into the compiler via a pipe?  But that would require quite a bit of extra java code.  It's currently written in a fairly simple way, where all the .ino / .pde files are turned into a single (temporary) .cpp file, before the "Compile" step is even begun.  While compiling, gcc is run using pretty much the same code with works on a paradigm of files being compiled to produce other files.  If the preprocessed output were never written to disk, not only would the preprocessing code need to be changed to more complex code to manage the whole thing in memory, but the compile code would need to run gcc differently for one part where the input isn't a file, and the entire structure within Arduino would need change to pass the in-memory copy between the "Sketch" and "Editor" sections and the "Compile" object.

That would be a monstrous patch, only to avoid writing one temporary .cpp file!



Quote

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code form a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.


I have often considered writing a patch to fix this.  It doesn't look like anyone else is ever going to do it....

I'd probably just reuse the existing parser that preprocesses the .pde / .ino files to extract a list of the #includes from all .cpp files, as if they'd been in the user's sketch.  But if there's something special it should do, please enlighten me?


bperrybap


But the code on github, soon to become Arduino 1.0.1, has a patch I wrote many months ago which avoids recompiling code.  It works more or less like you described, except it doesn't retain the .o files or a .a library for each board setting.  It simply uses the .o files left over in the temporary directory.

This feature has existed in Teensyduino for nearly 1 year.  Very soon it will be in Arduino.

Yep, I've been using it for quite some time.(I liked the promotional video).
I'm very happy to see the Arduino team finally accepted it for the standard release.

Quote

Any proposal to store compiled archives must carefully consider the restrictive shared network filesystems which are used by many academic institutions.  A patch which returns Arduino to the "bad old days" of writing within its own directories is certain to be rejected.


Compiled archives could live in the temporary build area.
This would essentially be a further optimization of of what Teensyduino was doing.
It would create an library archive for each board type as it is used and then
when possible re-use it when switching between board types or sketches.
Every time a new IDE session is started, since it gets a new build area, then the process starts clean.
But it would allow the library archive to expand during the IDE session.
For example, since the build directory does not change when you change sketches, then the core.a
and the library{boardtype}.a files would still exist and any libraries already compiled for that board
(even for a previous sketch during that IDE session) would not need to be compiled again.

Or even simpler, the IDE could build all the libraries for a board into an archive
inside the build area the fist time a sketch was was built.
It would be similar to the way the core library is built, only smarter.
Then, no library files would need to be rebuilt during that IDE session for that board.
In other words pay the library build time only on the first build (which would be a bit longer since it would build all libraries)
but then for the life of that IDE session builds are much faster even when you change sketches.


Quote

The library and core source files are never copied to the temporary build directory.  Only the sketch source is copied, and for good reason...


I stand corrected on this. (Not sure how I overlooked that)
I had thought all the full tree for each library library was copied, but it does not appear to copy the sources.
It does seem to create a "utility" directory in each library directory even when it is not used for the library.
Does the IDE currently use the .d files it creates?
(I assume it must use them with the Teensyduino "smart build" update?)

Quote

Quote

More importantly, as the Dr pointed out in the OP,
the way the IDE works, you are stuck having to add includes to the main sketch
in order to get the include path setup to allow other modules to compile.

And that is the main thrust of the the Dr's comment in the OP.
It was that in order for a module to use any
code from a library, the main sketch must include a header file from that library
to get the IDE to do some "magic" things and set up the include path.


I have often considered writing a patch to fix this.  It doesn't look like anyone else is ever going to do it....

I'd probably just reuse the existing parser that preprocesses the .pde / .ino files to extract a list of the #includes from all .cpp files, as if they'd been in the user's sketch.  But if there's something special it should do, please enlighten me?


That really worries me, because I've run into several cases where perfectly valid C/C++ code in a sketch
will not get past the IDE parser. I've had to alter the sketch code to get it to build.
There are also cases where the IDE appears to have some magic code to allow setting some defines inside the sketch.
It appears that because of this it looks for things and where to insert the includes.
In many of my sketches, it often screws up where to place the includes especially if there are ifdefs.
There are also simple things like a missing double quote on a string that will cause the IDE parser to crash
and you get no error message like you would with the real compiler making the error difficult to locate.
I'd be ok with sharing the parser with the library code but I think the mainline parser needs some
updates before I'd feel comfortable running all the library modules through it.

The "something special" might be that if a library referenced another library then that referenced library
would need to be added to the list of libraries. But maybe that falls out when the same parser is used.

--- bill

pjrc

#20
Mar 28, 2012, 08:28 pm Last Edit: Mar 29, 2012, 12:21 pm by Paul Stoffregen Reason: 1

Compiled archives could live in the temporary build area.
This would essentially be a further optimization of of what Teensyduino was doing.
It would create an library archive for each board type as it is used and then
when possible re-use it when switching between board types or sketches.


It already does reuse the compiled library and core files across multiple sketches.  But reuse between board changes is not done.  Teensyduino also treats a USB Type or CPU Speed change as a board change.

I don't intend to work further compiled file reuse, partly because the remaining cases are well into "diminishing returns", and partly because of the long process to get patches accepted.  Further work would probably require very substantial changes, which would have very little chance of being accepted into Arduino.


Quote

Or even simpler, the IDE could build all the libraries for a board into an archive
inside the build area the fist time a sketch was was built.
It would be similar to the way the core library is built, only smarter.
Then, no library files would need to be rebuilt during that IDE session for that board.


It already reuses .o files for libraries.

For example, if your sketch uses Servo and Wire, on the first compile the sketch, Servo, Wire and core files are compiled.  Thereafter, only your sketch is recompiled (unless you happen to edit the libraries or core with a text editor...)  If you add another include in the sketch to start using Stepper, on the next compile, Stepper is compiled, but Servo, Wire and the core files are all reused.

Sure, there are lots of other ways it could work, like pre-emptively compiling all the possible libraries.  But this approach makes minimal changes to the structure of Arduino's Compiler object and overall build process.  Even that has taken half a year to get accepted, or well over a year if you count an earlier version I submitted but was rejected.



Quote

Does the IDE currently use the .d files it creates?
(I assume it must use them with the Teensyduino "smart build" update?)


Yes.  There is a release candidate available, so you can try it on regular Arduino boards.  Here are the files:

Mac OS X: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-macosx.zip
Windows: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-windows.zip
Linux (32-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux.tgz
Linux (64-bit): http://files.arduino.cc/downloads/arduino-1.0.1-rc1-linux64.tgz
Source: http://files.arduino.cc/downloads/arduino-1.0.1-rc1-src.tar.gz

Of course, if you have Teensyduino, this is used for Teensy even on 0022 and 0023.

If you delete any of the .d files from the temporary build directory, it will force that .o file to be rebuilt.  If you haven't already, please give the release candidate a try.

Quote

Quote

I'd probably just reuse the existing parser that preprocesses the .pde / .ino files to extract a list of the #includes from all .cpp files, as if they'd been in the user's sketch.  But if there's something special it should do, please enlighten me?


That really worries me, because I've run into several cases where perfectly valid C/C++ code in a sketch
will not get past the IDE parser.


I'm imagining using the existing parser only to read the .cpp files, not modify them.  If it gets confused, the worst case result is a #include is not seen.  At least that that's the idea.... but I honestly have not yet worked on this, so I can't say for sure.

Quote

The "something special" might be that if a library referenced another library then that referenced library
would need to be added to the list of libraries. But maybe that falls out when the same parser is used.


Yes, recursive scanning might be needed if a library depends on another which in turn depends on others.

Circular include dependency might need to be detected and handled gracefully (not an infinite loop until the JRE eats up all available RAM).

If anyone really feels strongly about this, the java code is all open source.  This issue has been talked about over and over, but ultimately it's only going to be solved when someone actually does the work and submits a patch.  Anyone with some java skill can do this, since the parser, list of files and compiler object are already there.  It's "just" a matter of parsing all those .cpp files and adding to the existing lists, with the caveats mentioned above, and maybe other minor details I've glossed over or not anticipated.

This is issue 236, by the way.  If anyone does create a patch, that's the place to post it.

http://code.google.com/p/arduino/issues/detail?id=236

Since it's already a well recognized issue, odds are pretty good such a patch might actually find its way into Arduino in a fairly short time (eg, under 1 year).



Go Up