Go Down

### Topic: Encountered lousy boo-boo in arduino IDE generated compilation commands (Read 6657 times)previous topic - next topic

#### liuzengqiang

##### Mar 25, 2012, 06:50 pmLast Edit: Mar 26, 2012, 10:19 pm by liudr Reason: 1
You can't include a contributed library in a .cpp file without including a/any contributed library in the main .ino/.pde file!

I made simple tests by saving Blink into a regular sketch (not example any more) and adding a test.cpp tab to the file. I then added #include <phi_interfaces.h> in test.cpp but not in Blink.ino so this happens:
Code: [Select]
C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp.o C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp.o test.cpp:1:28: error: phi_interfaces.h: No such file or directory

Of course, the IDE failed to have this option on test.cpp!
Code: [Select]
-IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces

I then proceeded to add #include <phi_interfaces.h> in Blink.ino and this happens:
Code: [Select]
C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega -IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\Blink.cpp.o C:\Users\Liu\Downloads\arduino-1.0\hardware\tools\avr\bin\avr-g++ -c -g -Os -Wall -fno-exceptions -ffunction-sections -fdata-sections -mmcu=atmega1280 -DF_CPU=16000000L -DARDUINO=100 -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\cores\arduino -IC:\Users\Liu\Downloads\arduino-1.0\hardware\arduino\variants\mega -IC:\Users\Liu\Documents\arduino sketchbooks\libraries\phi_interfaces C:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp -oC:\Users\Liu\AppData\Local\Temp\build4240670370917479535.tmp\test.cpp.o

Notice the difference in the library locations included? If I didn't include that contributed library in Blink.ino the IDE fails to see the need for this contributed library location in any .cpp files. If I do include the contributed library in Blink.ino, the IDE slaps the contributed library location to all file being compiled. This is lousy at best.

#### bperrybap

#1
##### Mar 26, 2012, 05:29 am
I've fought this issue for quite some time. It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

My opinion and preference would be to modify the IDE so that
the users personal library directory as well as the Arduino distribution directory is

That way any code could include headers from any library by simply referencing the library's directory.

Example:
#include <Wire.h>
it would be:
#include <Wire/Wire.h>
For spi it would be:
#include <SPI/SPI.h>

etc...
This would solve the issue you are seeing and make it really easy for libraries to reference other classes in other libraries,
which today really doesn't work unless the user adds the include files to his sketch,
which makes no sense since the sketch doesn't need it
but it does have the side effect of adding the specific library's directory to the include path.

If the existing goofy include path stuff was still done, but these two additional directories were added,
then naive users could continue to do their header includes the way they are now,
and those developing libraries or additional modules could use the library directory references to include
the needed headers without the users sketch having to include the headers.

It would be a very small backward compatible transparent change to the IDE that would add new
functionality going forward for those developing more complex s/w like libraries that depend
on or use other libraries.

--- bill

#### pYro_65

#2
##### Mar 26, 2012, 05:52 am
This mildly frustrates me too, the reason you have to do this is because there is no option to add additional include directories ( As you have found ).

I have other needs when it comes to changing the command line call to gcc.
I have been thinking of writing a module for my server app that will intercept all the IDE's interaction with the file system, then modify the avr-gcc.exe call with my new parameters. If I continue the project I'll forward you the finished app.
Forum Mod anyone?
https://arduino.land/Moduino/

#### bperrybap

#3
##### Mar 26, 2012, 06:04 am
You could try using the mpide GUI to build the AVR sketches rather than the Arduino IDE.
mpide has additional capabilities in the boards.txt file
and introduces a platforms.txt which allows you to set custom defines and compiler
options on a per platform (AVR, pic32 etc..) or per board level within the platform.

There are also some decent makefiles out there for Arduino these days so you can bypass the IDE
completely.

--- bill

#### pYro_65

#4
##### Mar 26, 2012, 06:12 am
Quote
You could try using the mpide GUI to build the AVR sketches rather than the Arduino IDE.

I might give this a try, I would like to test some features of c++11 on the Arduino ( variadic templates, would revolutionize my library ).

http://arduino.cc/forum/index.php/topic,96976.0.html

I might plug through it one day with Google translator.
Forum Mod anyone?
https://arduino.land/Moduino/

#### liuzengqiang

#5
##### Mar 26, 2012, 06:18 am
Bill,

Yeah! It is OK for casual programmers especially those that don't know the difference between pde/ino and cpp

I've seen one possible solution to use #include <..\Wire\Wire.h> so if arduino team makes this change to all contributed library "import library", this may solve all the problem, will it?

pYro_65,

That'll be great! If you can intercept the commands and reassemble stuff to them, let us know!

At the moment I am developing user interfaces with LCD and keypads. I don't actually need any arduino hardware and simulated all necessary hardware (LCD, RTC, etc.) on PC I only occasionally need to teach arduino IDE how to compile by including libraries in the main sketch (the main sketch doesn't need all those libraries as Bill mentioned). All my work has been on Dev C++. I can say whether to look in subfolders for include files. I don't know if it is an IDE thing or there is a g++ option to look inside subfolders. Dev C++ has a horrible debugger. I can't change variable values although it says I can. Addresses are often misinterpreted so I can't see what my strings look like. Plus the variable view doesn't update itself. I'm ready to move on to a different IDE. Man. Arduino is good but maybe it's my time to move my largish project to some better IDE. Any suggestions for a good PC-based IDE that uses gcc (for my simulator) and a good arduino-compatible avr-gcc IDE?

#### bperrybap

#6
##### Mar 26, 2012, 06:46 am

I've seen one possible solution to use #include <..\Wire\Wire.h> so if arduino team makes this change to all contributed library "import library", this may solve all the problem, will it?

I tried using the relative path technique before on libraries.
It's been a while, and I recall having some kind of issue.
Like maybe it never built and pulled in the other library. (I can't remember)

Make sure to stay away from backwards slashes. Regular slashes is what should be used.
Normal slashes will always work in all OS environments - It is the C standard.
Backward slashes in file names/paths will only work in windows environments
that have special code to detect it.

--- bill

#### pYro_65

#7
##### Mar 26, 2012, 07:21 am
Arduino is good but maybe it's my time to move my largish project to some better IDE.

I have had a good experience with the Arduino IDE, my project is currently 14 files long and 165kb in size and growing, the IDE has been able to handle it all. With access to the gcc command line I will have all I need. I have been meaning to contact you about your library in regards to a common serial interface for HW and SW systems ( you had a thread a while back I think ).
Forum Mod anyone?
https://arduino.land/Moduino/

#### PaulS

#8
##### Mar 26, 2012, 02:56 pm
Quote
It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

#### bperrybap

#9
##### Mar 26, 2012, 05:53 pm

Quote
It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

It is not a nightmare. Finding and resolving references is what the linker does.

Libraries are supposed to be independent entities whose modules can be compiled and archived into the library
archive independently.
By creating a system that requires an application to set up things a certain way in its source code
in order to be able to compile a library module the independence is broken.
This defeats the entire point of having a library. It is no longer really a library at that point.
And that is why having the include path used for compiling a library module be determined
by the sketch/application is a totally brain dead thing to do.

The way this should work (and does in other environments) is that library modules are compiled
and archived into a library archive independently from application code.
An application includes only the headers that *it* needs.
The app is compiled and then linked against library archives and any library functions needed
are pulled in from the archive by the linker.
If a library function called by the application needs another library function,
The application does not and should not have to be aware of the additional library dependencies
of the library functions the application uses.
Imagine how painful things would be if you had to do this include stuff to set up things in "normal" applications.
Can you imagine how many headers you would have to include in your windows app code and how long
it would take to build everything?

The arduino system has a system of extensible libraries. If the IDE and build process were done differently
then all the modules in each library would be built once (or whenever a library module was updated/added)
and archived into a library vs being built every single time the sketch is compiled.
Then a sketch would include only the headers it needs.
The sketch would be linked against the Arduino library archive and the needed references would be resolved.
It would not matter if the references were from the sketch or from a library module that the sketch pulled in.

The problem is that the IDE is simply not treating the libraries as a proper library.
If it did, then these kinds of interdependencies would go way and sketch building would
be much faster since only the modules that changed would need to be re-compiled and archived rather
than having to build every single module used by the sketch and its libraries.

--- bill

#### liuzengqiang

#10
##### Mar 26, 2012, 06:34 pm

Quote
It is totally brain dead to base the include path for everything
based on the headers "noticed" in the users sketch, particularly when it comes to libraries.

I disagree with this. The advantage to listing all libraries in the sketch is that it makes it easy to see, looking at a sketch what libraries it needs.

If a sketch only needed to show that it needed library A, and library A needed libraries B, C, and D, and library C needed libraries E, F, and G, and library E needed 6 more levels of libraries, finding all the needed libraries would be a nightmare.

As it is, a sketch must list all libraries that it needs, so one can see at a glance what libraries will be needed.

One library can NOT hide the fact that it is dependent on another library.

Paul,

It's not just libraries. If I created a .cpp file and include a contributed library header, arduino IDE doesn't properly include that library's path when compiling the .cpp
There are times when I want to create classes in .h and .cpp but not necessarily make them arduino library. Say the classes are specific to only one project and other projects can't benefit from having the class in a library. Maybe I'm still working on the class and would keep it in a sketch folder so I can modify it in arduino IDE. There's more chances I can find to justify a better way of library links. I disagree with you on the fact the library should include all includes. That will be a very long list. I consider it better off if each library takes care of what it needs instead of what it's include needs.

#### PaulS

#11
##### Mar 26, 2012, 07:28 pm
Quote
It is not a nightmare. Finding and resolving references is what the linker does.

Finding and resolving references to installed libraries, yes. But the linker can't tell me what library I don't have installed.

If I copy a sketch posted on the forum/web/bathroom wall, the sketch currently must list all libraries needed. I like this feature. There is no need to install all the libraries listed in the sketch, only to find out that I need 14 more libraries, and then I need 7 more, and the 3 more, and then one more.

Quote
It's not just libraries. If I created a .cpp file and include a contributed library header, arduino IDE doesn't properly include that library's path when compiling the .cpp

The Arduino IDE makes a list of files to copy. That list includes all .ino/.pde files in the current directory, all files included by the sketches, and any .cpp files in the current directory that go with any header files in it's list. Then, it adds source files from libraries folders for header files that it does not yet have source files for.

When the list is complete, everything gets copied to one directory, and compiled there. There are no paths involved in the build process.

#### liuzengqiang

#12
##### Mar 26, 2012, 08:30 pm

When the list is complete, everything gets copied to one directory, and compiled there. There are no paths involved in the build process.

Paul, yes I agree to this fact. I've seen the temp folder and know what are in it. What I was saying in my OP was that not every file copied to the same temp folder had the right -I option for g++ to even compile. The linker comes in after everything compiles but not everything is compiled when arduino IDE tells g++ to compile without proper header path.

#### bperrybap

#13
##### Mar 26, 2012, 08:32 pm

There are no paths involved in the build process.

I think you need to have another look at how the build process works.
The library header files are included from their location in the "libraries" directories.
(user and system) by using an include path. This requires handing include paths to the compiler.

And that is one of the problems. The current build process requires that a separate include path
be given to the compiler for every single library used and those include paths are "learned" by
scanning through the users primary sketch file.
This is not a good way of doing things as it creates problems like what the Dr is seeing
and also keeps library modules from compiling that use other libraries unless the primary sketch
file has the additional, and unneeded by the sketch, header includes added to its source in order
to trick the IDE into modifying the include path to allow all the other modules to be able to locate

While I wouldn't do that way, it is ok to scan the users sketch file to learn of libraries to try to make things
simpler for the users sketch file. Where I have the problem is requiring the user to modify his sketch file in order to get
library or other modules to compile.

--- bill

#### liuzengqiang

#14
##### Mar 26, 2012, 08:57 pm
From my observation this is what I learned:

1) The sketch files in the sketch folder are compiled with path to their required headers in alphabetic order.
2) Then standard and contributed library .cpps are built in the order their headers show up in the main sketch. This includes things like LiquidCrystal or my own library.
3) Then the arduino core library .cpps are build in an unknown fashion, probably in the order they were enumerated in a file or mentioned in arduino.h. This includes all the wiring stuff, hardware serial, etc.

All above compiled .o files reside in the same temp folder as Paul mentioned, including the source file in step 1. After that some linker options (I can't understand fully) to mention all the .o file to be linked.