ANSWERED: Example of making a library using out-of-tree source files

robmeades · September 10, 2021, 12:10pm

Hi: stoopid newbie [to Arduino] question I'm afraid. We have a pre-existing source tree of multi-platform (ESP-IDF, nRF5 [GCC/SES], STM32F4, Zephyr] code in which the C source files are quite separate, "out of tree", from the metadata that builds those source files. In other words there is no src directory inside the SDK directory structure; there can't be because the code is intended to be compilable using [wildly!] different SDKs.

Since ESP-IDF is delivered as part of the Arduino tool-chain, and we support ESP-IDF, we have the opportunity to build our code with Arduino, allowing customers to use both our plain-old C code APIs and the Arduino APIs side by side.

However, I haven't found an example of the Arduino library format in which the source/header files are scattered around in variously named directories, ones that are NOT inside the library directory tree.

Can anyone point me at an example of how to do this?

in0 · September 12, 2021, 6:49am

Welcome, and thanks for your interest in Arduino!

Way cool!

I think it wouldn't be easy to accomplish this. The only possibility I can think of is symlinks, but the resulting library would be since symlinks are restricted on Windows and prohibited in Arduino Library Manager.

I think the best way to accomplish this is by writing a repackaging script that converts the repository to a series of valid Arduino libraries. Something like this has been done for TensorFlow lite and Edge Impulse. You could publish the repackaged libraries to dedicated repositories periodically, and distribute the libraries from those repositories to the Arduino users.

The resulting folder structure of the BLE library would look something like this:

u_ble
|-- src
|   |-- utility
|   |   |-- u_ble_cfg_extmod.c
|   |   |-- u_ble_cfg_intmod.c
|   |   |-- u_ble_data_extmod.c
|   |   |-- u_ble_data_intmod.c
|   |   |-- u_ble_extmod.c
|   |   |-- u_ble_intmod.c
|   |   |-- u_ble_private.c
|   |   `-- u_ble_private.h
|   |-- u_ble.h
|   |-- u_ble_cfg.h
|   |-- u_ble_data.h
|   `-- u_ble_module_type.h
|-- LICENSE
|-- README.md
`-- library.properties

You can define dependencies in the library.properties metadata file and Arduino Library Manager will offer the dependencies when the user installs the library.

The feature requested here sounds like it would avoid the need to either break the repository up into multiple libraries or rewrite the #include directives to use local paths:

github.com/arduino/arduino-cli

Allow Libraries to Extend Include Path

opened 08:24PM - 19 Nov 19 UTC

oclyke

type: enhancement

It would greatly benefit library developers if libraries could specify additions… to the include path used in their compilation. Developers could then create work that is much easier to maintain. ## Problem In my personal use case I would like to write both a portable C driver for a device (that can be used on other platforms and with minimal overhead) and an Arduino wrapper for that interface that makes it simple to use for beginners. For the purpose of illustration it would be best to think of the portable driver as immutable source code obtained from a third-party. For maintainability the third-party code would be included 'symbolically' (e.g. git submodule). A representative library structure might be: ``` bash libraries/Servo/library.properties libraries/Servo/keywords.txt libraries/Servo/src libraries/Servo/src/Servo.h # this is the Arduino wrapper interface libraries/Servo/src/Servo.c libraries/Servo/src/open-source-servo-driver-lib/include/ossdl.h # this is the portable open-source interface libraries/Servo/src/open-source-servo-driver-lib/src/ossdl.c ``` The file ```ossdl.c``` will most likely include the open-source portable interface with a line such as: ```#include "ossdl.h"``` -- unfortunately this will not work in Arduino 1.8.10 **The workaround(s) is(are) not great:** * **Maintain a Copy of OSSDL:** Increases mainenance workload, detracts from open-source spirit (collaborating on original source rather than forking and never coming back) * **Use Relative Paths in OSSDL.c:** ```#include "open-source-servo-driver-lib/include/ossdl.h"``` inexplicably associates the OSSDL project with Arduino build requirements * **Roll OSSDL into Arduino Library** (If you are the maintainer of OSSDL) Same issues as maintaining a new copy, and if you abandon the standalone copy then other potential users have to work around additional code (the Arduino implementation) ## Proposed Solution I suggest that library.properties gains a new comma-separated field of extensions to the library's include path (relative to ```Servo/src```). If not present (as in the case of existing libraries) only ```Servo/src``` will appear in the search path - keeping compatibility with the existing codebase. For the sake of interface protection this is as good as the current technique because users would have to modify ```library.properties``` to get access beyond the developer's original intent. Of course if they are modifying the library they don't need this feature to accomplish what they seek. To fix the example situation from above the developer would add the following entry to his ```library.properties``` file: ``` include=open-source-servo-driver-lib/include ``` Of course it would also be valid to use ``` include=lib1/include,lib2/include,lib2/src ``` since commas are not (typically) used in filepaths. If needed another format (or use of escape characters) could be employed. ## Other Possible Solutions (thoughts?) * **Linking Precompiled Libraries:** Although this may work in some cases it is counter-productive to educational (and open-source) efforts not to mention that users will lose control of compilation options * **Recursive Search for Library Headers:** This is the more extreme version of what is proposed - all ```.h``` files underneath the ```src/``` directory would be accessible. Concerns of interface preservation/enforcement would be raised. ## Related Issues / Feature Requests arduino/arduino-builder/issues/15 - A long-debated feature in which it is requested to put the Sketch folder on the include path for libraries. This is not the same request because * It keeps the library developer in control of the API * Does not add complexity for the typical user

robmeades · September 12, 2021, 8:59am

Thanks for the detailed response. Pity about not being able to refer to out of tree files from the library metadata but the link you provided gave me an idea: could Git submodule come to the rescue here?

Could we create a sister repo to our ubxlib repo named ubxlib_arduino or some such, which pulls in ubxlib under src? Can one specify a set of sub-directories of src in the library metadata, or does it always want to eat them all up (because there would be things in there related to other platforms etc. that it should not compile)?

in0 · September 12, 2021, 11:21am

Git submodules are not supported by Arduino Library Manager (sort of a package manager for Arduino Libraries which is integrated into the IDE, web editor, and Arduino CLI). Their presence in the repository won't block the acceptance of a library into the Library Manager index, but there will only be an empty folder where the submodule will be, just the same as when you use GitHub's "Download ZIP" button. Of course it's not mandatory that your library be in the Library Manager index, but it is a very convenient way to distribute the library and its updates to the users, so worth at least giving some consideration.

It recurses through everything under src and compiles all the source files.

There isn't any way to exclude paths from this recursion. This was actually the main reason Edge Impulse had to set up a repackaging script for their library. You can see here they requested an exclude capability:

github.com/arduino/arduino-cli

Feature request: exclude files from Arduino CLI

opened 10:10AM - 23 Mar 20 UTC

janjongboom

type: enhancement

I was contemplating whether this would be a request for the library format or so…mething in Arduino CLI. We have a unified codebase for our embedded libraries, and deliver a configuration file per target that includes / excludes certain folders. E.g. we can deliver a Makefile, a Keil uVision project, or an `.mbedignore` file as part of the delivery mechanism. This works really well. However, for Arduino there does not seem to be a similar way to exclude files from the build process. I know others struggle with this too (e.g. @neil-tan when developing uTensor). Adding this in the library format and in Arduino CLI would make it a lot easier to distribute upstream versions of libraries that we publish, rather than having to create separate builds just for Arduino.

robmeades · September 12, 2021, 10:12pm

Understood, a repackaging script it is. Thanks for the very helpful answers, I will go away and see what I can do now.

system · January 10, 2022, 10:12pm

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.