Per project libraries in IDE 2.3.6?

[mburnicki]

Hi all,
I'm working as software developer and have just recently tried my first Arduino project with Arduino IDE 2.3.6 appimage on Linux. The project works as expected so far, but I have a question regarding the handling of libraries by the IDE.
Actually, it looks like I can create an individual subdirectory for each project which effectively, but all libraries I import via the IDE are installed inthe same, global "libraries" directory.
IMO this becomes confusing if I have several projects which share different sets of libraries, and possibly even different versions of a specific library.

At work, I keep the full source code including local copies of the required libraries in a git repo, so I can always checkout a particulatr version of a project, including the appropriate versions of the library files. I don't see a way to do this with the Arduino IDE. If I have a earlier project built with a specific library file version, and then update the library due to a requirement of another project, I don't even know whether my older project works with the updated library, and I have no clue (from git) which version of the library file had originally been used with the older project.

So my goal would be a dircetory layout like

~/Arduino/Myproject1/MyProject1.ino
~/Arduino/Myproject1/libraries/...

~/Arduino/Myproject2/MyProject2.ino
~/Arduino/Myproject2/libraries/...

on only use the libraries below the project folder, so I can create a git repo in every project folder (MyProject1, Myproject2, ...) which contains all the stuff needed to rebuild a particular version of the project.

Yet, I haven't found a way to achieve this. Whenever I import a library using the IDE, it's installed in the common directory, and if I move the "libraries" directory into a project folder, the compiler doesn't find it.

Is this possible, and if yes, can someone give me a clue how to achieve this?

[sonofcy]

While I could argue against your approach, the way that I have heard about to do what you want is to use VSCode and PlatformIO. It supports per project libraries.

[mburnicki]

Thanks. I’m going to have a look at that and give it a try. I’m usually working with VSCode but have not yet heard about PlatformIO. I guess I’ll have to manually install the libraries I need for a project, rather than managing them via the IDE, right?

BTW, I’d really be interested to know what you might argue agains my approach, except that libraries may need to be maintained manually. IMO, tracking both project-specific files and the specific versions of library files used to build a particular program version is the only way to maintain programs in a clearly traceable manner.

[Deva_Rishi]

I have several versions of the same libraries in my libraries folder of my sketchbook.

All that requires is that i rename them, an put them in a folder with a different name (usually the same as the header file i renamed) and that i modify the files that include that header file to include that renamed file (and the #ifdef BLABLA_h #define BLA_BLA_H if that's in there.

That way any modifications i have made to that specific version remain untouched. Cores get updated and therefore certain libraries don't work anymore or libraries get updated and don't work on an older core etc. etc.
And i include the specific library header file i want to use.

There is the option of simply putting the library files in the same folder of the .ino files, but i do usually have multiple projects (or versions of the same project) that share a library version, so i don't do it like that.

[vvb333007]

@mburnicki

can someone give me a clue how to achieve this?

not in arduino ide.

you can’t pass linker flags to libraries let alone having different libraries for different projects.

Visual Code or Visual SlickEdit

[ptillisch]

Hi @mburnicki. The Arduino IDE developers are tracking the request for support for per-sketch dependencies management here:

Use configuration from sketch project file

If you have a GitHub account, you can subscribe to that thread to get notifications of any new developments related to this subject:

---

Please only comment on the GitHub issue thread if you have new technical information that will assist with the resolution. General discussion and support requests are always welcome here on the Arduino Forum.

---

Arduino has already implemented this capability in the official Arduino CLI tool. So if you don't mind working with a command line tool, you should definitely take a look at that option:

https://arduino.github.io/arduino-cli/latest/sketch-project-file/#build-profiles

@mburnicki

all libraries I import via the IDE are installed inthe same, global "libraries" directory.

If you want to use Library Manager to manage the dependencies on a per-sketch basis, you could use a dedicated sketchbook folder for each project. Library Manager installs libraries to the libraries subfolder of the sketchbook, and the compiler will use the libraries from that location.

This means you would need to set Arduino IDE's "Sketchbook location" preference to the appropriate location for the given sketch you are working with (or you could also accomplish it by directly editing the configuration file where that preference is stored if you prefer). I know this is not a very convenient solution, but thought it was at least worth mentioning.

@mburnicki

I keep the full source code including local copies of the required libraries in a git repo, so I can always checkout a particulatr version of a project, including the appropriate versions of the library files. I don't see a way to do this with the Arduino IDE.

The Git repository would be the sketch's dedicated sketchbook folder. The folder would contain the sketch and its library dependencies.

@mburnicki

my goal would be a dircetory layout like

~/Arduino/Myproject1/MyProject1.ino
~/Arduino/Myproject1/libraries/...

With my solution it would instead be:

.../Myproject1/Myproject1/MyProject1.ino
.../Myproject1/libraries/...

where .../Myproject1/ is the project's dedicated sketchbook folder, and the root of the project's Git repository.

@mburnicki

if I move the "libraries" directory into a project folder, the compiler doesn't find it.

Arduino IDE compiles all the source files located under the src subfolder of the sketch:

https://arduino.github.io/arduino-cli/latest/sketch-specification/#src-subfolder

So you can bundle libraries with a sketch by placing them under a subfolder named src. The folder is recursed, so you are welcome to organize the libraries in any arbitrary folder structure under that specific folder.

If you take this approach, you do need to adjust the #include directives in the sketch to relative paths. For example, if your sketch contains this line:

#include <Foo.h>

And you bundled the library that provides that header file like so:

Myproject1/
├── Myproject1.ino
└── src/
    └── libraries/
        └── FooLib/
            ├── library.properties
            └── src/
                ├── Foo.h
                ...

Then you would need to change the line to this:

#include "src/libraries/FooLib/src/Foo.h"

In some cases you might also need to adjust the #include directives in libraries (example scenario described here). If you run into that problem and would like assistance, just let us know.

---

The Arduino IDE developers are tracking the request for Arduino IDE to recognize libraries bundled under a libraries subfolder of the sketch in the same way the global libraries are installed (avoiding the need to adjust #include directives) here:

Support bundling libraries with a sketch

@mburnicki

have not yet heard about PlatformIO. I guess I’ll have to manually install the libraries I need for a project, rather than managing them via the IDE, right?

No. PlatformIO has its own library management system that is equivalent to the Arduino IDE Library Manager.

@mburnicki

I’d really be interested to know what you might argue agains my approach, except that libraries may need to be maintained manually.

In the case where it is the only viable solution, it is fine.

However, in the case where there are alternatives, the argument against this "vendoring" approach it is it can cause the repository size to blow up. Some libraries contain tens or even hundreds of megabytes of supplemental binary files.

The alternative approach is to instead only check a "manifest" file into the repository. This file defines all dependencies of the project, and is used by the tooling to install those specific dependencies. This is how the Arduino "build profiles" system works, and also how it works with PlatformIO's system. It is also the standard practice in non-Arduino projects using popular dependency management systems such as Go modules, npm, and Poetry.

The downside of that "manifest" approach is that you are relying on the continuing availability of the dependencies from an external source, which makes it possible that the project could be broken by a dependency being removed from the package manage, and also potentially more vulnerable to a supply chain attack.

[mburnicki]

Thanks for the idea. But a preferred method would be to (optionally ?) store required libraries locally inside a project and be able to manage (e.g. upgade) them on a per project base.

[mburnicki]

Thanks!

[mburnicki]

Thanks for the pointer to github. I’ve subscribed to that topic.

Thanks also for the thought about the pros and cons of different approaches to handle this. As I said, I’m still very new to using the Arduino environment, with 2 tiny projects, so I can try several methods and see what fits best.