Using library in another library

Hi,

I want to use an arduino library (SoftwareSerial) inside my own library.
I noticed that i need to include SoftwareSerial.h inside the sketch, or else i can't include it in my code.
I guess that the IDE generates compilation flags (include path) based of the content of the sketch?

My problem with that is that i find it very counterintuitive for the user to need to include a second library, because support for SoftwareSerial is optional in my library.
It is ok that a user must include SoftwareSerial if he use it, but if someone want to use my library with HardwareSerial only, he would still need to include SoftwareSerial in the sketch ...

My question is thus : is there any workaround to fix this?

Subquestions:
-Is there any way (using preprocessor or other) to detect at compile time if SofwtareSerial has been included in the sketch?
I could then use conditionnal compilation to disable SoftwareSerial support in my library if SoftwareSerial is not included in the sketch.
-Is there any way to include a library header in a library even if it is not included in the sketch? (apart from giving the full absolute path)

Thanks in advance for your help

I want to use an arduino library (SoftwareSerial) inside my own library.

Permission granted.

I noticed that i need to include SoftwareSerial.h inside the sketch, or else i can't include it in my code.

Correct, for very good reasons.

I guess that the IDE generates compilation flags (include path) based of the content of the sketch?

It copies files to the build directory based ONLY on the sketch contents.

My problem with that is that i find it very counterintuitive for the user to need to include a second library,

As the Eagles say "Get over it".

It is ok that a user must include SoftwareSerial if he use it, but if someone want to use my library with HardwareSerial only, he would still need to include SoftwareSerial in the sketch ...

Then there is a problem with how you are making the use of SoftwareSerial optional.

-Is there any way (using preprocessor or other) to detect at compile time if SofwtareSerial has been included in the sketch?

When the SoftwareSerial source file is compiled? No. What you need to do is make sure that the linker knows that it doesn't need to link anything from the SoftwareSerial library.Then. it won't matter that the SoftwareSerial library was not copied/compiled.

-Is there any way to include a library header in a library even if it is not included in the sketch? (apart from giving the full absolute path)

Not if you want the header and source files for that library to be copied/compiled.

Thanks for your answers, PaulS.
I am sure there are very good reasons for this library compilation behavior, i just try to figure out how to use it in my case.

Actually, the problem is not at link time, but at compile time, when i include the header SoftwareSerial.h, the compiler does not find it if it is not included in the sketch.

btw, i am not sure if it would be considered a bug or just a quirk of the IDE, but if i include SoftwareSerial in the sketch, and then remove it (without changing the library code), it compiles, because the IDE does not recompile my library.

Tyrel,
Yes you are correct. The IDE uses a very primitive and dumb methodology.
In my view, this was done from ignorance and lack of understanding of how real embedded development is done.
Because of their build methodology, the IDE doesn’t truly support libraries using other libraries.
Yes, the existing hack of including the sub library’s header in the main sketch allows things to work but it was not really designed to work this way. While it has been enhanced to better handle sub libraries over the years, In reality it is a mere accident/artefact of the way the IDE searches for sketch libraries and the way it handles it’s libray include paths.
There are better ways to handle this but the Arduino team is pretty stuck on using the current build methodology.

Yes you can include a header for another library in your library. It can be done by referencing from a library that included.
i.e. #include<…/…/…/libname/headername.h>
That will include the header and allow your code to build, the problem is that because the IDE is using such a primitive build methodology, it won’t compile or link in the sub library.
For example, you know that the AVR core library (the only true Arduino library - which is all the built in functions like digitalWrite() ) is always on the include path when building for AVR, you can reference
SofwareSerial relatively from that.
i.e
#include <…/…/…/…/libraries/SofwareSerial/SoftwareSerial.h>
(I think I got the relative path correct for the latest IDE - but you get the idea)

-Is there any way (using preprocessor or other) to detect at compile time if SofwtareSerial has been included in the sketch?

Usually. You have to take advantage of detecting some information created by the dumb build methodology currently used by the IDE along with some broken compiler/pre-processing handling that exists in the newer versions of gcc.

The combination of the two allow you to do this.

I do this in some of my libraries. I detect that the user hasn’t included a header that my library needs and crash the compiler with an error saying as much.

Essentially, the way this works, is that the newer versions of gcc are not properly handling #include when the target file cannot be found. Some gcc versions consider it ok to include a file that doesn’t exist and treat that ok/acceptable and just silently ignore this error and keep going like there is no issue.
Some versions spit out a warning and keep going (you won’t see them because the Arduino IDE blocks all warnings!)
The latest gcc seems to treat it as an error but appears to be so paralleled and pipelined from cpp to cc processing that they keep going for quite some time trying to continue to try to compile the code even though the #include failed. It eventually does stop with a #include error but you get lots of funky warnings/errors as well.

Regardless of which behavior you have, you can take advantage of the compiler processing your code beyond a missing #include to emit your own error.

With respect to the Arduino IDE, this works because if the sketch does not included the header for the needed sub library, then the include path will not contain the directory for the sub library. If the include path does not contain this sub library directory, then the #include in your library code will fail to locate the sub library header.

What you can do, is in your source that includes the needed sub library header, you then use a preprocessing check to see if a define exists for the sub library immediately after the include for that library.
If you don’t see the define, crash the compiler with an error.
This works because gcc continues to process code for a while/beyond a #include with a missing target file.

I have never understood why gcc does this. To me this is very broken behavior. (gcc didn’t do this going back to 1989 until just a few years ago) But you can use this behavior to your advantage when using the wimpy arduino IDE and its dumb build methodology to try to give the end user a clue as to how to get your library working.

i…e
in your code add:

#include <SoftwareSerial.h>
#ifndef SoftwareSerial_h
#error Sketch must include SoftwareSerial.h
#endif

This will crash the compiler with your error if the SoftwareSerial.h header file can’t be found.
Because of the dumb Arduino Build methodology it won’t be found unless the user has included it in his sketch since the SofwareSerial library directory won’t be on the include path without the user including it in his sketch.

tyrel:
Thanks for your answers, PaulS.
I am sure there are very good reasons for this library compilation behavior, i just try to figure out how to use it in my case.

There are other better ways this could have been handled. I believe it started out as a quick and dirty way of handling it and it has now ballooned into a monster that is actually as complex if more than doing it the “normal” way REAL libraries are handled in s/w development.

Actually, the problem is not at link time, but at compile time, when i include the header SoftwareSerial.h, the compiler does not find it if it is not included in the sketch.

As shown above, you can either force the include or crash the compile depending on what you want/need.
I have had a need to both.
All this ugliness is because of the poor build methodology the IDE uses.
If you leave the IDE behind, it can be done correctly, and there are some makefile templates out there that handle libraries correctly. The problem is that if you are creating a library for other Arduino users, most of them are not very technically sophisticated and so they will struggle with using something other than the IDE, particularly users that are still using Windows since Windows wasn’t designed for s/w development and hence doesn’t come set up with many of the needed and useful tools out of the box like the *nix operating systems do.

btw, i am not sure if it would be considered a bug or just a quirk of the IDE, but if i include SoftwareSerial in the sketch, and then remove it (without changing the library code), it compiles, because the IDE does not recompile my library.

Perhaps you mean link rather than compile?
I would expect that if you removed the header from the sketch, the compile for your library should fail because the needed sub library directory would not be on the include path.

So I’m guessing that when you remove the header file, the sketch recompiles and everything links using the pre-built objects like your previously compiled library objects.
It could be considered a bug, this is intertwined with their build system.
Because they abandoned having the IDE using makefiles long ago for all dependency checking, they have now tried to do all of what make does inside the IDE. This combined with the way they handle libraries is a real mess and you get subtle issues like this one.

— bill

Wow, thanks for the extensive answer!
It seems that you don’t like the current build methodology much ;-).
Out of curiosity, is there any documentation about how it works exactly, because the official documentation is rather vague?
It would probably help if more developpers could understand how it work, why it works this way, and how/why it should be improved.

For now, i think i will go with the ugly #include <…/…/…/…/libraries/SofwareSerial/SoftwareSerial.h>, and leave the user with undefined reference error if they use the SoftwareSerial option of my library without including it.
That seems an acceptable solution.

Your other solution is not exactly what i want since i do not always want a error if the user does not include SofwareSerial.
However, i have learned something about how gcc handle not found headers… Really strange, but afaik, the C++ standard does not specify what the compiler must do in that case, so it is unspecified behaviour.

I will probably give a try to the makefile option someday, but indeed, for public release, this is not a viable option.

For the IDE, yes, it seems that the library is not recompiled if only the sketch is modified (which would make sense in most build systems, but not here). I will submit a bug report.

tyrel:
Wow, thanks for the extensive answer!
It seems that you don’t like the current build methodology much ;-).
Out of curiosity, is there any documentation about how it works exactly, because the official documentation is rather vague?
It would probably help if more developpers could understand how it work, why it works this way, and how/why it should be improved.

I’ve not seem much internal Arduino documentation. From my perspective, Arduino is pretty much a hack job all the way around. It is very “pretty” and usable on the outside, but pretty ugly on the inside, especially the IDE.
The thing that frustrates me the most is that Arduino tends to have a not invented here syndrome going on. Not nearly as bad as it was in years past but there is so much re-inventing the world going on.

For now, i think i will go with the ugly #include <…/…/…/…/libraries/SofwareSerial/SoftwareSerial.h>, and leave the user with undefined reference error if they use the SoftwareSerial option of my library without including it.
That seems an acceptable solution.

Not sure how your library is structured.
I was involved with another library that had multiple classes in it. And because of the dumb way the IDE works, all the class modules for that library had to be in the same Arduino “library” directory which means that they all had to be built - regardless of whether they were actually used. This forced things like library class code that used the Wire library to be built even the code eventually wasn’t linked into the users sketch.
It was all transparent to the user since all they needed to know was that if they were using the class that used the Wire interface they had to include the Wire.h header but if using some other interface they left it out.
They could choose to always include the Wire header but if they included it and didn’t use the class the used the Wire interface, the IDE would still build and link in the IDE code. The way the Wire code is written the Wire code becomes dead code in the image if the sketch includes the Wire header even if no code uses or references any Wire routines. While related and cause by the IDE build methodology this Wire “dead code” issue could be solved by changing some of the Wire code.

Needless to say, it was ugly and all because of the IDE build methodology.
In that case I had to use the relative include path to allow the code to always be compilable.

Your other solution is not exactly what i want since i do not always want a error if the user does not include SofwareSerial.
However, i have learned something about how gcc handle not found headers… Really strange, but afaik, the C++ standard does not specify what the compiler must do in that case, so it is unspecified behaviour.

Yeah, seems to be C++ knuckled head issue. There have been some heated discussions about it in the past and that was back when gcc (actually the C++ preprocessor) was changed to support not only ignoring missing include headers but to also recognize and support including the same header multiple times when the header doesn’t have guard defines.
(Code at one point was added to try to detect and silently handle multiple includes of the same file as well as a recursive loop of includes)
To me there was no need to warp the pre-processor and compiler to handle peoples broken development environments.
All the reasons given for the support of this go back to broken code and broken development environments.
The gcc maintainers should have given them a big FU and told them to back and fix their broken environments rather than warp gcc to work around their lame issues.
In the worst case, if they wanted to start mucking with behavior, they should have added commandline options to control the behavior. That way everyone could have had the behavior that they wanted without forcing anything on anybody.
At least a missing header file has gone back to being an error vs being silently ignored or just spitting out a warning.

For the IDE, yes, it seems that the library is not recompiled if only the sketch is modified (which would make sense in most build systems, but not here). I will submit a bug report.

I’ll be curious as to their response. Not sure how far you will get since what you are doing falls outside of the way they intend the IDE and libraries to be used. It is also one very contrived example.
Because of this, I tend to think that the existing behavior probably needs to stay as it is.
In the normal/typical case of development with the IDE, you would not want to recompile the world when a sketch changes since it should not impact any of the existing pre-compiled library code.
What is the alternative?

There is no real way to know that there is a dependency between a pre-compiled library module and a user sketch, since technically each should be separately compilable.
In your library, both the sketch and your code are separately compilable; with the issue being that you lose your include path to the sub library when you remove the include for the sub library from the sketch.
This is a side effect of the way they handle Arduino “libraries” (which are not real libraries) but I’m not sure I’d want to rebuild everything just because the sketch code changed - which normally is not necessary.

— bill

I’ll be curious as to their response. Not sure how far you will get since what you are doing falls outside of the way they intend the IDE and libraries to be used. It is also one very contrived example.

As expected , the issue has been closed rapidly, saying that it is a known limitation of the IDE.
However, they also directed me to a recent discussion about the possibility to improve the dependence resolution in the new command line compilation tool ‘arduino-builder’, so all hope is not lost :wink:

Yes you are correct. The IDE uses a very primitive and dumb methodology.
In my view, this was done from ignorance and lack of understanding of how real embedded development is done.

Interesting. I actually want to do "real embedded development", but I don't have bucket loads of money to spend. How can I do that? Can you suggest a tool chain? I'm aware of Atmel Studio 7, and looking into it. It's been a while, but I have programmed in a "normal" dev environment using C. Are there any "gotchas"?

I don't think this question creates any conflict of interest here because so few users would be interested in working with that level of complexity.

If you don't like the Arduino build system, the don't use the Arduino build system. The language is plain old c++, so you are free to do your own builds, using makefiles, and you can do the build any way you like. If you create your project with .cpp files, rather than .ino files, then the Arduino build process is essentially bypassed. Nick Gammon has a whole page on his excellent site on exactly how to get around the quirks of the Arduino build process.

Regards,
Ray L.

aarg:
Interesting. I actually want to do "real embedded development", but I don't have bucket loads of money to spend. How can I do that? Can you suggest a tool chain? I'm aware of Atmel Studio 7, and looking into it. It's been a while, but I have programmed in a "normal" dev environment using C. Are there any "gotchas"?

The gcc toolset isn't the issue. It is the Arduino IDE that has the limitations.
There are alternatives to using the Arduino IDE.
When not using the IDE the biggest gotchas will be that the IDE does some massaging that allows sketches to avoid doing some of the normal required things for C/C++ like prototype declarations.

Since I've been doing embedded coding since the early 80's and working with gcc since 1989,
I lean to using hand written makefiles whenever I want to do anything serious.
I also use *nix operating systems and avoid using Windows since Windows was not designed for s/w development like unix was and lacks many tools that come out of the box on *nix systems.
Yeah, there are ports to Windows, but Windows is just so hostile to the commandline.

--- bill

bperrybap:
I also use *nix operating systems and avoid using Windows since Windows was not designed for s/w development like unix was and lacks many tools that come out of the box on *nix systems.
Yeah, there are ports to Windows, but Windows is just so hostile to the commandline.

— bill

Or you can use Cygwin, or other similar tools, and have virtually all of the standard *nix tools under Windows…

Regards,
Ray L.

RayLivingston:
Or you can use Cygwin, or other similar tools, and have virtually all of the standard *nix tools under Windows....

Regards,
Ray L.

That is what I meant by "Yeah, there are ports to Windows"
While I have used them and they do work. It isn't as good as the real thing.
Keep in mind I've been using Unix for over 30 years and did lots of kernel and driver development for more than 2 decades so to me, things like MingW and Cygwin tools are frustrating because even with that, you still don't have all the tools, debuggers, scripting and i/o capabilities that come standard with a *nix OS.
i.e. you still have to deal with Windows and its limitations even if using things like MingW and Cygwin tools since that is the underlying OS.

--- bill

Back to the original topic, according to a new answer to my bug report (Problem in build process when library include another library · Issue #3877 · arduino/Arduino · GitHub), the library-to-library dependencies are now officially supported in the new build system.