ANSWERED: Example of making a library using out-of-tree source files

Hi: stoopid newbie [to Arduino] question I'm afraid. We have a pre-existing source tree of multi-platform (ESP-IDF, nRF5 [GCC/SES], STM32F4, Zephyr] code in which the C source files are quite separate, "out of tree", from the metadata that builds those source files. In other words there is no src directory inside the SDK directory structure; there can't be because the code is intended to be compilable using [wildly!] different SDKs.

Since ESP-IDF is delivered as part of the Arduino tool-chain, and we support ESP-IDF, we have the opportunity to build our code with Arduino, allowing customers to use both our plain-old C code APIs and the Arduino APIs side by side.

However, I haven't found an example of the Arduino library format in which the source/header files are scattered around in variously named directories, ones that are NOT inside the library directory tree.

Can anyone point me at an example of how to do this?

Welcome, and thanks for your interest in Arduino!

Way cool!

I think it wouldn't be easy to accomplish this. The only possibility I can think of is symlinks, but the resulting library would be since symlinks are restricted on Windows and prohibited in Arduino Library Manager.

I think the best way to accomplish this is by writing a repackaging script that converts the repository to a series of valid Arduino libraries. Something like this has been done for TensorFlow lite and Edge Impulse. You could publish the repackaged libraries to dedicated repositories periodically, and distribute the libraries from those repositories to the Arduino users.

The resulting folder structure of the BLE library would look something like this:

u_ble
|-- src
|   |-- utility
|   |   |-- u_ble_cfg_extmod.c
|   |   |-- u_ble_cfg_intmod.c
|   |   |-- u_ble_data_extmod.c
|   |   |-- u_ble_data_intmod.c
|   |   |-- u_ble_extmod.c
|   |   |-- u_ble_intmod.c
|   |   |-- u_ble_private.c
|   |   `-- u_ble_private.h
|   |-- u_ble.h
|   |-- u_ble_cfg.h
|   |-- u_ble_data.h
|   `-- u_ble_module_type.h
|-- LICENSE
|-- README.md
`-- library.properties

You can define dependencies in the library.properties metadata file and Arduino Library Manager will offer the dependencies when the user installs the library.

The feature requested here sounds like it would avoid the need to either break the repository up into multiple libraries or rewrite the #include directives to use local paths:

Thanks for the detailed response. Pity about not being able to refer to out of tree files from the library metadata but the link you provided gave me an idea: could Git submodule come to the rescue here?

Could we create a sister repo to our ubxlib repo named ubxlib_arduino or some such, which pulls in ubxlib under src? Can one specify a set of sub-directories of src in the library metadata, or does it always want to eat them all up (because there would be things in there related to other platforms etc. that it should not compile)?

Git submodules are not supported by Arduino Library Manager (sort of a package manager for Arduino Libraries which is integrated into the IDE, web editor, and Arduino CLI). Their presence in the repository won't block the acceptance of a library into the Library Manager index, but there will only be an empty folder where the submodule will be, just the same as when you use GitHub's "Download ZIP" button. Of course it's not mandatory that your library be in the Library Manager index, but it is a very convenient way to distribute the library and its updates to the users, so worth at least giving some consideration.

It recurses through everything under src and compiles all the source files.

There isn't any way to exclude paths from this recursion. This was actually the main reason Edge Impulse had to set up a repackaging script for their library. You can see here they requested an exclude capability:

Understood, a repackaging script it is. Thanks for the very helpful answers, I will go away and see what I can do now.