I want to store my libraries locally inside my projects, so that I can easier build it on different PCs without having to install the libraries on each PC.
I understand that the arduino-cli looks inside the documents\arduino folder and probably also AppData\local\arduino15, but they are too global for me. I want to have the libraries in version control together with my project. Is there any way I can instruct arduino-cli to look for libraries in folders relative to my project?
Hi @nielsdybdahl. The best solution will be to use a "build profile":
https://arduino.github.io/arduino-cli/latest/sketch-project-file/#build-profiles
With this solution, you don't actually put the source code of the libraries into version control, but instead the "sketch project file" that defines the dependencies of the sketch.
In case that doesn't meet your needs, you can configure the path where Arduino CLI looks for libraries via the directories.user configuration key:
An alternative solution is to specify an additional libraries location via the --libraries flag of the arduino-cli compile command:
https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/#options
Welcome to the forum
I know very little about arduino-cli but does it allow library files to be placed in the same folder as the .ino file of the sketch as is possible in the IDE ? They are then #included like this
#include "filename.h"
You can also use a relative path such as
#include "libraries\test.h"
where the libraries folder is in the same folder as the .ino file of the sketch
Yes. Anything that will work in Arduino IDE will also work with Arduino CLI.
If you take this approach, the libraries must be under the src subfolder of the sketch folder. Arduino CLI/Arduino IDE only compile the source files that are located under the src subfolder, so if you put the library under a different path, the header files will be #included, but the definitions in .cpp etc., files will not be compiled so it will only work in the rare case where the header files contain all necessary code.
The IDE (2.3.2) seems quite happy to #include a library file that is held in a folder named libraries in the sketch folder
ie, this works
main sketch
#include "libraries\test.h"
void setup()
{
Serial.begin(115200);
testPrint();
}
void loop()
{
}
test.h
void testPrint()
{
Serial.println("printed from test.h");
}
I found the solution: Adding a compiler option like the following works:
arduino-cli --libraries=libraries sketchfile
And then all libraries can be in the subfolder libraries.
The documentation states that the option is named builtin.libraries but that does not work
I too have wanted something like what @nielsdybdahl is asking about.
I REALLY wish that the build system would simply provide the capability of a local "libraries" directory under the sketch directory.
i.e. if the sketch directory has a "libraries" directory then that libraries directory would be used just like the sketchbook/libraries directory but have the highest priority in the library search order and on the compile -I include path.
That would make a very simple and fairly intuitive way for users to create a bundled project that included all the necessary libraries.
It would allow Arduino project developers to create something as simple as a zip file that could contain EVERYTHING that is needed to build a project.
It would also allow the entire project bundle - including all the needed libraries, to be source controlled as a bundle..
It would also be simple for the project users as all they would have to do is unzip the project and it could be built with the IDE.
I'm guessing something like this is what @nielsdybdahl is looking for.
An implementation like this would not be difficult to add to the Arduino build tools.
In fact, I proposed this years ago, but it went no where.
--- bill
Or they could just put them in the sketch folder and use the alternative #include format of filename
The type of user that is likely to want/need such a facility would, I suspect, have no problem with that or with explicitly using the libraries\filename.h style having put the library files in place
Yes, as I explained:
Move the definition of testPrint to libraries/test.cpp and check what happens.
It seems a bit odd to say you "found" that, after I told you:
But the important part is that there is a solution.
Maybe you missed the part of my reply where I said to use the directories.user configuration option?:
The Arduino CLI developers are tracking the request here:
If you have a GitHub account, you can subscribe to that thread to get notifications of any new developments related to this subject:
There is a draft of the work for adding the requested capability here:
If you have a GitHub account, you can subscribe to that thread to get notifications of any new developments related to this subject:
Please only comment on the GitHub issue thread if you have new technical information that will assist with the resolution. General discussion and support requests are always welcome here on the Arduino Forum.
Sure but it REALLY clutters up the actual project.
Having a separate area for libraries that are outside the project is cleaner and avoids having to muck with the source code to support it or any potential name collisions on files.
Also, if a local libraries directory under a sketch was supported, if an advanced user wanted to use a newer/different version of the library vs the project bundled one, it is easier to just remove that particular library's directory and the sketch code could then automatically use a library installed by the library manager. No mucking with the sketch code would be required.
As an alternative, a project could be done as a library instead and specify the needed libraries as dependencies.
And then make the sketch of the project an example.
While this wouldn't actually bundle the needed libraries,
it makes the installation of the project and installing the needed library easier for the project users since the users could use the library manager to search for and install the project (library) and the IDE library manager would also install the needed libraries if they were specified as dependencies.
Updates to the project could also be handled by the library manager.
IMO, this is a fairly workable option and does provide quite a bit of flexibility with respect to the libraries used given capabilities of library dependencies.
It depends on what the primary goal is.
If it is creating a project including bundling libraries so that the project can supply a known working environment and ALL of that can be properly source controlled, there really isn't currently a way to do that with the Arduino IDE other than tossing everything (all the needed libraries) into the sketch sketch directory - which IMO, is a mess.
If the primary goal is make it easier for the users to install the project and get up and going, then making the project a library that is known to the library manager can do that and may suffice, particularly given the project library could specify the needed libraries so that the needed libraries can automatically be installed.
And in Arduino 2.x the project library could even specify the desired/supported version(s) of a needed libraries by using the version constraints capability of the depends property. So it is nearly the same as actually including the libraries and this method would provide some additional capabilities over bundling the actual libraries.
--- bill
General question for @ptillisch
I've not used version constraints
What happens when a library specifies version constraints on a dependent library?
How does the IDE library manager handle that?
(docs don't specify how it really works)
Is it only looking for and using this at top level library install time?
Or is it handled by the builder that will download and install that version of the dependent library somewhere it can be used during the build process.
If it is only handled at install time, then this really doesn't work since multiple libraries could be specifying different version constraints for the same dependent library.
And what happens if a version of the dependent library outside the constraint is already installed?
Is there an error installing the top level library or does the currently installed dependent library get clobbered by a version of the library specified inside the constraint?
If the IDE supports the ability for multiple libraries to specify conflicting constraints, then the IDE would need to have support for some area to hold these specific versions of the dependent libraries that don't interfere with other libraries dependencies.
something like a local libraries directory under the sketch might work for that but I haven't seen any indication that the IDE would be capable of support multiple libraries having different constraints on the same dependent library.
--- bill
The version problematic is what I handle by having the libraries stored with each project instead of having global libraries.
Your suggestion started with a "build profile" which looks promising, but it seems not to specify where the libraries are located, so it completely misses my problem and then I skipped reading the rest. Then I spent a lot of time trying to get the command line parameters from the documentation working until I found out that the documentation is outdated and the parameters are different now.
That is correct.
Please note that the topic of this discussion is Arduino CLI, not Arduino IDE.
This is how it works when a build profile is used:
https://arduino.github.io/arduino-cli/latest/sketch-project-file/#build-profiles
When using a build profile, an isolated set of project dependencies are used when compiling, so they aren't subject to changes to the globally shared libraries and boards platforms installed and updated normally via Library Manager and Boards Manager. This is the Arduino equivalent of the reproducible build frameworks provided by tools such as JavaScript's npm or Python's Poetry.
The behavior can be controlled by the user. For example, the 1.0.6 release of the "SunFounder AI Camera" library specifies a dependency on "ArduinoJson" version >=6.0.0:
By default, installing "SunFounder AI Camera"@1.0.6 will upgrade an existing installation of ArduinoJson@5.13.5:
$ arduino-cli lib install ArduinoJson@5.13.5
Downloading ArduinoJson@5.13.5...
ArduinoJson@5.13.5 ArduinoJson@5.13.5 already downloaded
Installing ArduinoJson@5.13.5...
Installed ArduinoJson@5.13.5
$ arduino-cli lib install "SunFounder AI Camera"@1.0.6
Downloading ArduinoJson@7.0.4...
ArduinoJson@7.0.4 ArduinoJson@7.0.4 already downloaded
Installing ArduinoJson@7.0.4...
Replacing ArduinoJson@5.13.5 with ArduinoJson@7.0.4...
Installed ArduinoJson@7.0.4
Downloading SunFounder AI Camera@1.0.6...
SunFounder AI Camera@1.0.6 SunFounder AI Camera@1.0.6 already downloaded
Installing SunFounder AI Camera@1.0.6...
Installed SunFounder AI Camera@1.0.6
However, if we don't want the dependencies installation to cause changes to any existing library installation, we can add the --no-overwrite flag to the arduino-cli lib install command:
$ arduino-cli lib install ArduinoJson@5.13.5
Downloading ArduinoJson@5.13.5...
ArduinoJson@5.13.5 ArduinoJson@5.13.5 already downloaded
Installing ArduinoJson@5.13.5...
Replacing ArduinoJson@7.0.4 with ArduinoJson@5.13.5...
Installed ArduinoJson@5.13.5
$ arduino-cli lib install --no-overwrite "SunFounder AI Camera"@1.0.6
Error installing SunFounder AI Camera: Library ArduinoJson@7.0.4 is already installed, but with a different version: ArduinoJson@5.13.5
That capability is provided by build profiles.
It doesn't miss. Instead, it takes a more reasonable approach to the problem. There is no need to check the source code of the project dependencies into your repository. The only thing you need is to check in a sketch project file that specifies the versions of the dependencies that should be used. This is a standard software development practice. For example when developing a JavaScript project, we would never check in the node_modules folder where the npm package dependencies are installed because all we need is the information in the package.json and package-lock.json files to replicate that.
That is not a very respectful way to behave when you are receiving free assistance.
Make sure to use the documentation available here:
https://arduino.github.io/arduino-cli/latest/commands/arduino-cli/
This is automatically generated from the Arduino CLI source code so it will always be up to date. If you are using a different version of Arduino CLI, you can select that version from the menu at the top of the page to get the documentation for the specific version you are using. If you are using the nightly build version of Arduino CLI, select "dev" from that menu.
The Sketch file structure example has a library, SomeLib, under the src directory
Foo
|_ arduino_secrets.h
|_ Abc.ino
|_ Def.cpp
|_ Def.h
|_ Foo.ino
// middle omitted
|_ src
|_ SomeLib
|_ library.properties
|_ src
|_ SomeLib.h
|_ SomeLib.cpp
Should the fact that the subdirectory has a library.properties allow that to be picked up as a "local library" automatically, so that you can do
#include <SomeLib.h>
without any extra library configuration; instead of the manual
#include "src/Somelib/src/SomeLib.h"
I tried it, and got during Detecting Libraries
Alternatives for SomeLib.h: []
ResolveLibrary(SomeLib.h)
-> candidates: []
fatal error: SomeLib.h: No such file or directory
#include <SomeLib.h>
^~~~~~~~~~
compilation terminated.
Unfortunately, no. There is no such feature at this time. As I mentioned in a previous reply, the developers are tracking a request for an equivalent feature (except using a dedicated subfolder instead of reusing the src subfolder):
Sorry I missed that previous link; thanks.
So if SomeLib has no special treatment, putting it in the Sketch file structure example like that is misleading?
Back to the original question: if you have some shared code among several sketches, and you're already using version control, then with Git (for example) you could use submodules, still under src so that code is picked up. The include is then
#include "src/OtherLib/OtherLib.h"
That is correct.
You should be aware that there are some specific cases where you might need to adjust #include directives in the library:
Correct Inappropriate #include Syntax
Let's say you have a sketch structure like this:
Foo/
├── Foo.ino
└── src/
└── SomeLib/
├── SomeLib.cpp
└── SomeLib.h
And SomeLib.cpp contains an #include directive for SomeLib.h. The appropriate syntax for that #include directive would be:
#include "SomeLib.h"
which causes the compiler to search the local path for the file. Unfortunately, many Arduino library developers use inappropriate syntax:
#include <SomeLib.h>
which causes the compiler's "search path" to be searched for the file. When the library is installed normally, they get away with it because the sketch build system automatically adds the library's source path to the search path, but it doesn't do that for libraries bundled with the sketch in this way, so under this scenario the library would fail to compile with a "no such file or directory" error.
So you might need to go in and change the angle brackets to double quotes if you are working with libraries written by sloppy developers. If you encounter it, it would be worth submitting a pull request to the library repository since this can be considered a bug in their code.
Use Relative Paths for Interdependent Libraries
Let's say you have a sketch structure like this:
Foo/
├── Foo.ino
└── src/
├── OtherLib/
│ ├── OtherLib.cpp
│ └── OtherLib.h
└── SomeLib/
├── SomeLib.cpp
└── SomeLib.h
And SomeLib.h contains an #include directive for OtherLib.h:
#include <OtherLib.h>
In this case, the library developer's choice of the angle bracket syntax is appropriate because OtherLib.h is not part of the "SomeLib" library. That works fine when the "OtherLib" library is installed normally because the sketch build system automatically adds the "OtherLib" library's source path to the compiler's search path. However, that won't be the case when the "OtherLib" library is bundled with the sketch in this way, so under this scenario compilation would fail with a "no such file or directory" error.
The solution is to modify such #include directives to use the relative path to the file, in this example:
#include "../OtherLib/OtherLib.h"
In a professional environment, you can not rely on resources hosted by third-parties. You need to have complete control on what libraries you use, their versions, what licenses they use and where they are hosted. We can either host them on our artifact server or in our version control system, but not with any third parties. How would I set up a build profile if I want to retrieve the libraries from our artifact server? (we use Artifactory)