conflicting library versions

nomadic68 · January 9, 2021, 5:21pm

Hi all,

I am currently using arduino-cli, and am writing to ask if it is possible to disable the 2 default library include paths
(in Windows):

C:\Program Files (x86)\Arduino\libraries
%userprofile%\Documents\Arduino\libraries

The reason is that for some of my projects i am using older versions of some libraries because with the latest ones my code doesn't compile anymore (library interface changed). I don't want to update my code to support the newest libraries because this does not protect me from the problem possibly happening again when a library gets updated further.

Instead, i would like to keep the relevant libraries inside the project source tree and use the --library option to include them; but if i do so, i get compile errors because the 2 directories above are always included by default, thus ending in conflicting/duplicate declarations.

A temporary (and horrible!) solution i found was to leave the default library directories empty by relocating the system and contributed libraries somewhere else, but this adds extra work to keep the libraries updated (for my newer projects) since the library manager doesn't work anymore.

An additional command line option to disable these 2 default locations or a CLI version requiring to mandatorily specify also the 2 aforementioned library paths would be perfect.

Does anybody know if there is a secret/hidden command option or configuration to alter to accomplish that?
Thanks and regards

pert · January 9, 2021, 6:19pm

nomadic68:
I am currently using arduino-cli

...

C:\Program Files (x86)\Arduino\libraries

Instead, i would like to keep the relevant libraries inside the project source tree and use the --library option to

This is not clear to me. Are you using Arduino CLI?:
https://arduino.github.io/arduino-cli/latest/
or the command line interface of the standard Arduino IDE?:

github.com

arduino/Arduino/blob/master/build/shared/manpage.adoc

// Generate a manpage with: a2x -f manpage manpage.adoc
// or HTML with: a2x -f xhtml manpage.adoc
//
// This file uses {empty}:: in some places, to allow putting multiple
// paragraphs inside a single label list item. This is a bit ugly and
// non-semantic, but it seems this is the best way to do this. Asciidoc
// also supports putting a plus sign on a line by itself to join two
// paragraphs into a single list item. However, the indentation on the
// second paragraph makes that formatted with a fixed-size font.
// Removing the indentation completely makes the asciidoc source very
// unreadable. Also, for the --board option, there is a a paragraph,
// followed by a list, followed by another paragraph. The + approach can
// only put the latter paragraph into the inner list, not the outer
// one...

ARDUINO(1)
==========
:doctype: manpage

NAME

This file has been truncated. show original

The reason I ask is because Arduino CLI only uses the libraries subfolder of the Arduino IDE installation folder in the event it is located in the Arduino IDE installation folder, which is an unusual circumstance at this time when Arduino CLI has not yet been integrated with Arduino IDE.

The other thing I don't understand is the mention of the --library option. Neither program has such an option. Arduino CLI does have a --libraries option.

nomadic68 · January 13, 2021, 4:53pm

Hi,

I am using the Arduino CLI, not the command line interface of the standard IDE:
my aim is to have a better control of the whole source tree (including libraries) and the compiler options.
(that is, like i had with Atmel Studio when it was working; at a certain point the support for the ARM boards broke and never got fixed, so i moved away from AS7 and landed to Visual Studio Code and Arduino CLI).

And yes, I was meaning the --libraries options of Arduino CLI, sorry for the typo.

Would it be possible to make a change request to the CLI developers to allow better control of the libraries folders? How?

Thanks

pert · January 13, 2021, 5:11pm

OK

nomadic68:

C:\Program Files (x86)\Arduino\libraries

As I mentioned before, this folder will only be used by Arduino CLI if it's bundled with the Arduino IDE at C:\Program Files (x86)\Arduino. This is in preparation for the planned complete replacement of arduino-builder with Arduino CLI as the engine behind the Arduino IDE's GUI. So you can simply use a standalone installation of Arduino CLI to avoid the use of this folder.

nomadic68:
2) %userprofile%\Documents\Arduino\libraries

Arduino CLI always uses the libraries subfolder of the user directory. %userprofile%\Documents\Arduino is the default user directory, but you are welcome to set it to any location you like. You can learn more about that here:
https://arduino.github.io/arduino-cli/latest/configuration/
Note also that configuring Arduino CLI has become even more convenient with the addition of the ability to control the configuration settings via arduino-cli config:
https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_config/

You can have a separate configuration file for each project if you need different settings.

nomadic68:
and the compiler options.

You can inject arbitrary compiler flags using the arduino-cli compile --build-property flag:
https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/#options
Some detailed discussion about that here:

github.com/arduino/arduino-cli

Provide dedicated sets of "extra flags" properties for platform developers and users

opened 03:04PM - 20 Jul 20 UTC

per1234

type: enhancement topic: documentation criticality: high topic: build-process

### Describe the request Establish two sets of standard "extra flags" build p…roperties: - one set for use by platform developer - one set for use by the user These should be clearly documented as being dedicated to the exclusive use by either the platform developer or the user. The properties proposed here are only arbitrary properties, without any special use by the build system, so no changes would required in the build system. It is only a matter of setting an example for 3rd party platform developers by adding these additional properties to the `platform.txt` files of the official platforms with comments clearly explaining their intent. Ideally, the properties would be documented in the [Arduino platform specification](https://arduino.github.io/arduino-cli/latest/platform-specification/) as well (https://github.com/arduino/arduino-cli/issues/985). ### Describe the current behavior Arduino has established a convention of providing "extra flags" properties in [`platform.txt`](https://arduino.github.io/arduino-cli/latest/platform-specification/#platformtxt). These properties are intended to allow the compilation process to be customized by the addition of arbitrary flags in the compilation commands. For example: https://github.com/arduino/ArduinoCore-avr/blob/1.8.6/platform.txt#L39-L49 ```text # This can be overridden in boards.txt build.extra_flags= # These can be overridden in platform.local.txt compiler.c.extra_flags= compiler.c.elf.extra_flags= compiler.S.extra_flags= compiler.cpp.extra_flags= compiler.ar.extra_flags= compiler.objcopy.eep.extra_flags= compiler.elf2hex.extra_flags= ``` These properties are referenced in the appropriate compilation recipes in `platform.txt` ([example](https://github.com/arduino/ArduinoCore-avr/blob/1.8.6/platform.txt#L58)). The empty definitions in `platform.txt` ensure that if the extra flags are not defined elsewhere, the compilation recipes referencing them will still work. Those default empty definitions can be overrriden from any of the other platform configuration files: - [`boards.txt`](https://arduino.github.io/arduino-cli/latest/platform-specification/#boardstxt) - [`boards.local.txt`](https://arduino.github.io/arduino-cli/latest/platform-specification/#boardslocaltxt) - [`platform.local.txt`](https://arduino.github.io/arduino-cli/latest/platform-specification/#platformlocaltxt) - [`global platform.txt`](https://arduino.github.io/arduino-cli/latest/platform-specification/#global-platformtxt) or from the command line: - [`arduino-cli compile --build-property`](https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/#options) - [`arduino --pref`](https://github.com/arduino/Arduino/blob/master/build/shared/manpage.adoc#options) --- From the comments in the official `platform.txt` files, it seems that `build.extra_flags` is intended to be for the use of the boards platform developer, while the other "extra flags" properties are intended to be for the use of the user (because a platform developer has no need for `platform.local.txt`). A common use case of these properties for the user would be to add `-D` flags (e.g., https://github.com/arduino/arduino-cli/issues/210#issuecomment-493892258, https://github.com/arduino/Arduino/issues/6344), but there are other uses. However, platform developers sometimes use these properties in their `boards.txt` file ([example](https://github.com/MCUdude/MightyCore/blob/v2.0.5/avr/boards.txt#L103-L111)). This presents the problem that the user risks inadvertently overriding the flags defined by the boards platform if they use the "extra flags" properties to modify the compilation command ([example](https://github.com/arduino/arduino-cli/issues/2041)). ### Arduino CLI version N/A ### Operating system All ### Operating system version Any ### Additional context #### Re: Possible Harmful Effect on APIs On a certain level, this does provide the user the global macro setting capability that has been often requested, and often rejected: - https://github.com/arduino/arduino-builder/issues/15 - https://github.com/arduino/arduino-builder/pull/29 - https://github.com/arduino/arduino-builder/pull/282 - https://github.com/arduino/arduino-builder/issues/283 - https://github.com/arduino/arduino-builder/issues/354 - https://github.com/arduino/arduino-ide/issues/1837 - https://github.com/arduino/Arduino/issues/421 - https://github.com/arduino/Arduino/issues/2547 - https://github.com/arduino/Arduino/issues/3635 - https://github.com/arduino/Arduino/pull/3717 - https://github.com/arduino/Arduino/pull/3757 - https://github.com/arduino/Arduino/issues/4528 - https://github.com/arduino/Arduino/issues/5084 - https://github.com/arduino/Arduino/issues/5390 - https://github.com/arduino/Arduino/issues/7189 - https://github.com/arduino/Arduino/issues/11108 - https://github.com/arduino/arduino-cli/issues/159 - https://github.com/arduino/arduino-cli/issues/210 - https://github.com/arduino/arduino-cli/pull/1117 - https://github.com/arduino/arduino-cli/pull/1517 - https://github.com/arduino/tooling-rfcs/issues/9 - https://github.com/arduino/tooling-rfcs/issues/22 However, I think this proposal strikes the right balance of: - Allowing advanced users to do advanced things - Not exposing standard users to additional complexity that makes the Arduino development software less beginner friendly - Being very easy for Arduino to implement The concern has been that allowing users to set global macros will result in library developers unnecessarily using these as part of the API, making that API less beginner friendly (https://github.com/arduino/arduino-builder/issues/15#issuecomment-145558252). However, if the user is required to use a command line interface or create a specially formatted configuration file in a difficult to access location, the average user won't be interested, and so it will only make sense for a library developer to use this feature for advanced, rarely used features of the library (e.g., enable debug output for troubleshooting of the library). We have already had this "extra flags" capability [since 2014](https://github.com/arduino/ArduinoCore-avr/commit/2b9f022eaad0d107b1516c22ac02bdac4977223b), yet I haven't seen it result in any significant incorporation of user set global macros into the user-level API of Arduino libraries. ### Issue checklist - [X] I searched for previous requests in [the issue tracker](https://github.com/arduino/arduino-cli/issues?q=) - [X] I verified the feature was still missing when using the latest [nightly build](https://arduino.github.io/arduino-cli/dev/installation/#nightly-builds) - [X] My request contains all necessary details

nomadic68:
Would it be possible to make a change request to the CLI developers to allow better control of the libraries folders? How?

Sure, though it seems to me that the tool already provides all the functionality you seek. You can learn about submitting feature requests and bug reports here:
https://arduino.github.io/arduino-cli/latest/CONTRIBUTING/#issue-reports

nomadic68 · January 14, 2021, 1:23pm

@Pert: 1000 thanks for the useful indications, now i have a lot to work on!
You are right, i have downloaded the Arduino-CLI and started using it without uninstalling my Arduino IDE, so the CLI found the boards and the libraries automatically. I will look at the configuration options you pointed me to in order to understand them better.
I also came across the PRO version of the IDE, i got curious, downloaded it and gave it a look: surprisingly, the PRO IDE didn't find the normal IDE folders. If i get this one to work the way i want, it will save me from the hassle of studying how to set up VS Code instead. Wow!

pert · January 14, 2021, 1:39pm

nomadic68:
@Pert: 1000 thanks for the useful indications, now i have a lot to work on!

You're welcome. Please let me know if you end up having any other questions or problems.

nomadic68:
i have downloaded the Arduino-CLI and started using it without uninstalling my Arduino IDE, so the CLI found the boards and the libraries automatically.

There's no problem with having Arduino CLI and Arduino IDE installed at the same time. I always have both installed (as well as Arduino Pro IDE). By default, Arduino CLI does use the same user, data, and download directories as the Arduino IDE, but you can always configure it to use other directories if that's a problem for you. The only time the Arduino IDE being installed has special implications for Arduino CLI is if Arduino CLI is installed to the same folder as the Arduino IDE, but that's easy enough to work around simply by moving Arduino CLI somewhere else.

nomadic68:
If i get this one to work the way i want, it will save me from the hassle of studying how to set up VS Code instead. Wow!

I'm glad if the Arduino Pro IDE provides for your needs. Arduino Pro IDE actually uses Arduino CLI under the hood so you will find the Arduino CLI configuration file information also applies to the Arduino Pro IDE configuration.

nomadic68 · April 5, 2021, 7:24pm

Hi Pert/all,
Sorry for the long silence, sometime work becomes crazy...

I tried both the arduino-CLI and the new IDE 2.0 and found something different than what Pert wrote in his replies to my question; to be absolutely sure, i repeated the tests in a clean virtual machine (win10 x64) and got the same results; my installation is identical on the real PC and the virtual machine:

Arduino IDE 1.8.13 installed using Windows installer (not the marketplace app!), all default directories.
Added SAM and SAMD boards packages.
Added Adafruit MCP23017 library just to have one (installed via library mgr it ended in the sketchbook library folder).
Arduino-CLI 0.18.0 in the same directory of 1.8.13 IDE
Arduino IDE 2.0 beta 4 in C:\Arduino2IDE_b4

The outcome of the tests:

Arduino-CLI:
I didnt install any boards nor libraries from Arduino-CLI.
the CLI lists the same boards as the IDE 1.8.13 (correct)
the CLI lists only the custom library (MCP23017) i installed from the IDE; the "system" libraries (SD, SPI, Ethernet...) are not there. Running the "lib update-index" command doesnt change the result (different from Pert wrote)

Arduino 2 IDE:
it sees the same boards as IDE 1.8.13 (correct)
when i click on "include library" in the new empty sketch, it sees only: SPI HID SoftwareSerial EEPROM Wire as "system" libraries, and the MCP23017 user library.
when i install Ethernet library it gets installed in the sketckbook library folder.

These are really from 1.8.13 IDE because if i clean the virtual machine and run CLI or IDE 2.0 without prior installation of the 1.8.13 IDE i see no boards and no libraries at all.

I also tried to download the Ethernet library from the CLI, and it has been installed in the sketchbook libraries folder (%USERPROFILE%\Documents\Arduino\libraries).

Is there something broken?

PS: i gave a try also to the new Microchip Studio: it still has the same problems as the old Atmel Studio about compiling for Due.

Best regards

pert · April 5, 2021, 8:09pm

nomadic68:
the "system" libraries (SD, SPI, Ethernet...) are not there. Running the "lib update-index" command doesnt change the result (different from Pert wrote)

The libraries you have mentioned are actually of two different types. You might be used to always having SD and Ethernet pre-installed because they come with the installation of the classic Arduino IDE, but neither Arduino CLI nor Arduino IDE 2.x come with any pre-installed libraries, so you must install these.

As for SPI, this is a platform bundled library. By default, arduino-cli lib list does not show these libraries. But it will if you add the --all flag to the command. You can also add the --fqbn flag to show only the platform bundled libraries for that specific board. Documentation here:
https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_lib_list/

With that cleared up, is there anything else that is not as you expect?

nomadic68 · April 6, 2021, 3:00pm

Hi Pert, thank you for your quick reply which clarified my misunderstanding of the Arduino IDE organization.

If i remember well, with a previous version of the arduino-cli in the same directory as the IDE executable i was getting the IDE pre-installed libraries listed even without the --all flag.
Never mind: the current behavior is perfect for me, because at the beginning of the thread i was asking if there was a way to exclude the 2 default directories, that is:

c:\Program Files (x86)\Arduino\libraries
%USERPROFILE%\Documents\Arduino\libraries
in order to have a more AS7-like management of my projects, with a local copy of all required libraries in the project folder (still have to find how to get rid of 2) folder, though).

Sorry if i have irritated you, it is clear that i missed to read the relevant parts of the online documentation, relying only on the help text.

BTW, is there a way to get an offline copy like PDF or similar, so i can read it thoroughly while i travel?

Thanks and regards!

pert · April 6, 2021, 4:02pm

nomadic68:
If i remember well, with a previous version of the arduino-cli in the same directory as the IDE executable i was getting the IDE pre-installed libraries listed even without the --all flag.

That was indeed the way Arduino CLI worked two years ago, but it was changed to the current behavior in the 0.3.7 release.

nomadic68:
the current behavior is perfect for me

I'm glad to hear it!

nomadic68:
Sorry if i have irritated you, it is clear that i missed to read the relevant parts of the online documentation, relying only on the help text.

Not at all. I love talking about this stuff, probably a lot more than most people have any desire to hear about it.

nomadic68:
BTW, is there a way to get an offline copy like PDF or similar, so i can read it thoroughly while i travel?

The source of the documentation you see hosted at https://arduino.github.io/arduino-cli/latest/ is here:

The "Command reference" section of the documentation is not there, and neither the "gRPC reference" section because those are generated from the Go code of Arduino CLI. But the Command reference is identical to the command line help you get from arduino-cli and there's a good chance you have no interest in the gRPC interface.

Even though it's the raw markup for the documentation in those files in the docs folder, Markup is quite readable even in its raw state, and there are also plenty of applications that will render it for you, VS Code's Markdown preview for example.

If you want to get really fancy, you can serve a local copy of that website from your own computer. The instructions for that are here:

https://arduino.github.io/arduino-cli/latest/CONTRIBUTING/#working-on-docs

You do need to install the tools, but after that, it's simply a matter of running the command task docs:serve from the folder containing the Arduino CLI repository. I use that system all the time while doing development on the documentation.

Even though this documentation is hosted in the Arduino CLI repository, much of the information applies to all the official Arduino development software (Arduino IDE, Arduino IDE 2.x, Arduino Web Editor). The reason is that Arduino CLI is used under the hood by all those applications to handle non-GUI tasks. It's definitely not something the average beginner should be exposed to, but for those of us who like to dive a bit deeper, there is some very interesting and valuable information there.

nomadic68:
Thanks and regards!

You're welcome. I'm glad if I can be of assistance.

klavorar · April 22, 2021, 1:37pm

btw I have been created a package manager for any Arduino project which takes care the versions also (so if any conflict is in place with dependencies it warns you etc..).
It's super simple to use but still makes your project mainntainable and portable!

You can find it here: GitHub - ksrichard/apm: Arduino Package Manager

system · August 20, 2021, 1:38pm

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Library path Programming	3	4904	May 5, 2021
multiple directories Programming	2	594	May 5, 2021
Resolved - Advice sought on multiple libraries causing update issues IDE 1.x	11	680	December 8, 2023
Possible library conflict Libraries	1	229	August 26, 2024
I need help with libraries and versions Programming	3	430	May 21, 2021

conflicting library versions

Related topics