IDE picks old version of installed 3rd party library after library upgrade

[alex_73]

Hi!

I am currently working on an ESP8266 based project in Arduino IDE v1.8.13.
I upgraded Streaming.h library to the latest version. As it seems I now have two versions of Streaming,h because during compilation I get

Code: [Select]

Alternatives for Streaming.h: [arduino_467138@6.0.8 arduino_412692@5.0.0]
ResolveLibrary(Streaming.h)
  -> candidates: [arduino_467138@6.0.8 arduino_412692@5.0.0]

Unfortunately the IDE picks version 5.0.0 instead of 6.0.8. (see last -I parameter):

Code: [Select]

"C:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\tools\\xtensa-lx106-elf-gcc\\2.5.0-4-b40a506/bin/xtensa-lx106-elf-g++"
-D__ets__ -DICACHE_FLASH -U__STRICT_ANSI__
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/include"
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/lwip2/include"
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4/tools/sdk/libc/xtensa-lx106-elf/include"
"-IC:\\Users\\alex\\AppData\\Local\\Temp\\arduino_build_979772/core" -c -w -Os -g -mlongcalls -mtext-section-literals
-fno-rtti -falign-functions=4 -std=gnu++11 -ffunction-sections -fdata-sections -fno-exceptions -w -x c++ -E -CC
-DNONOSDK22x_190703=1 -DF_CPU=80000000L -DLWIP_OPEN_SRC -DTCP_MSS=536 -DLWIP_FEATURES=1 -DLWIP_IPV6=0 -DARDUINO=10813
-DARDUINO_ESP8266_WEMOS_D1MINIPRO -DARDUINO_ARCH_ESP8266 "-DARDUINO_BOARD=\"ESP8266_WEMOS_D1MINIPRO\""
-DFLASHMODE_DIO -DESP8266
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4\\cores\\esp8266"
"-IC:\\Users\\alex\\AppData\\Local\\Arduino15\\packages\\esp8266\\hardware\\esp8266\\2.7.4\\variants\\d1_mini"
"-ID:\\CloudStorage\\Dropbox\\Work\\Arduino\\libraries\\arduino_485728\\src"
"-ID:\\CloudStorage\\Dropbox\\Work\\Arduino\\libraries\\arduino_412692\\src"
"C:\\Users\\alex\\AppData\\Local\\Temp\\arduino_build_979772\\sketch\\MAX7219_MQTT_IoTWebConf.ino.cpp"
-o nul -DARDUINO_LIB_DISCOVERY_PHASE
:
:
Multiple libraries were found for "Streaming.h"
 Used: D:\CloudStorage\Dropbox\Work\Arduino\libraries\arduino_412692
 Not used: D:\CloudStorage\Dropbox\Work\Arduino\libraries\arduino_467138
:


How can I tell the IDE to use the latest version or the version I intend for this particular sketch?
Thanks a lot,
Alex.

[PieterP]

Why are your libraries installed in Arduino/libraries/arduino_xxxxxx? Please see https://www.arduino.cc/en/guide/libraries#toc5.

As a side note, there's no need to use third-party streaming libraries on the ESP8266, it includes the standard library streams, so you can just use those:

Code: [Select]

#include <iostream>

void setup() {
  Serial.begin(115200);
  std::cout << "Hello, world" << std::endl;
}

void loop() {}


Pieter

[alex_73]

Why are your libraries installed in Arduino/libraries/arduino_xxxxxx? Please see https://www.arduino.cc/en/guide/libraries#toc5.

I did not install the libs via zip but IDE library manager search and also upgraded via the IDE library manager. I do not know why some libraries have the arduino-xxxxxx prefix and others don't. Most likely because the lib devs decided so?
The arduino sketch folder is also in a different location than the default (Preferences->Sketchbook location).

As a side note, there's no need to use third-party streaming libraries on the ESP8266, it includes the standard library streams, so you can just use those:

Thanks for the hint, I will have a look at this. One of the benefits of Streaming.h I see are the convenience macros for base conversion (_HEX), string padding(_WIDTHZ), ... makes life so much more comfortable.

Unfortunately this might solve the current problem with Streaming.h, the underlying issue is still open... I mean, worst case I still can delete the old version from disk. But that would be too easy, or?
Btw, I see this behavior also on other libraries like IotWebConf, ArduinoJson to name two of them.

I opened issue on github

[pert]

I did not install the libs via zip but IDE library manager search and also upgraded via the IDE library manager. I do not know why some libraries have the arduino-xxxxxx prefix and others don't. Most likely because the lib devs decided so?

No, the libraries should not be installed to folders with these names under normal circumstances. When you install a library via Library Manager, it is first installed to this randomly named folder, then the final step is for it to be named to the final folder name. The problem comes when there is already a folder of the same name. Normally, the IDE replaces that folder with the new library installation, hopefully upgrading the library. However, under some circumstances the IDE is unable to do that. In this case, the newly installed library remains in the randomly named arduino_xxxxxx folder. This usually happens for me when I have a file from that library folder open at the time of the Library Manager installation. I suspect it could also happen if an antivirus or perhaps cloud storage software interferes with the Library Manager installation process.

worst case I still can delete the old version from disk. But that would be too easy, or?

That's definitely what you should do.

How can I tell the IDE to use the latest version or the version I intend for this particular sketch?

You can learn about the factors that influence which library is picked when multiple libraries contain the #included file name here:
https://arduino.github.io/arduino-cli/latest/sketch-build-process/#dependency-resolution

In this case, there isn't a clear winner between the two so it comes down to one of these two:

5. A library that has a folder name with a better score using the "closest-match" algorithm wins
6. A library that has a folder name that comes first in alphanumeric order wins

I would expect that to be deterministic so I don't know what is the cause of the non-deterministic behavior regarding which is used that you reported on GitHub. Under normal circumstances, there is almost always a clear winner between the libraries.

[alex_73]

However, under some circumstances the IDE is unable to do that. In this case, the newly installed library remains in the randomly named arduino_xxxxxx folder.

Hmm, then I'd expect to see some error or at least some warning during installation which I did not. Installs went smooth in the past. So, would it be safe to rename the arduino_* folders to the filename of the main .h/.cpp?

That's definitely what you should do.

At first I hesitated because if someone on the issue ticket asks for some retest my testcases are different due to timestamps and possibly other things. But as it was hindering me now really badly, I moved the old versions out.

I would expect that to be deterministic so I don't know what is the cause of the non-deterministic behavior regarding which is used that you reported on GitHub. Under normal circumstances, there is almost always a clear winner between the libraries.

You are right, from the documentation the order should be always the same.
To be honest, I already hoped that ArduinoIDE finally provided a way for a user to select a specific version of a library :-) Oh was I wrong ...

[pert]

Hmm, then I'd expect to see some error or at least some warning during installation which I did not.

Yeah, I agree that it should notify the user about this situation, probably even halt or fail the installation and then remove that temporary folder if the lock couldn't be resolved. But unfortunately the IDE gives no indication at all when this happens.

At first I hesitated because if someone on the issue ticket asks for some retest my testcases are different due to timestamps and possibly other things. But as it was hindering me now really badly, I moved the old versions out.

That is a good sentiment. As someone who does a lot of beta testing of the Arduino software I often find myself in the same sort of situation; where my goal is to figure out how to reproduce an error, not to fix it. In the case of the arduino_xxxxx folders, it's easy to reproduce this simply by having a file open in the library when doing a Library Manager update of it.

However, at least from a support perspective it would be interesting to understand other potential causes of this issue. As I said, I have a hypothesis that antivirus on-access scanners/etc. could also cause it but I don't have any concrete evidence to back it. I only know that I only get this issue under the expected conditions but occasionally someone else reports regularly getting it  without any direct interference on their part. So if you end up learning more about what caused it on your system I would be interested to hear about it.

[alex_73]

However, at least from a support perspective it would be interesting to understand other potential causes of this issue.
:
So if you end up learning more about what caused it on your system I would be interested to hear about it.

I have a regular Windows 10 setup with stock Microsoft virus scanner.

BUT, as I did arduino dev in the past on different machines, my Sketchbook location is inside a Dropbox folder - for obvious reasons ;-)
I temporarily pointed the folder to a different location not monitored by dropbox and the folders were named nicely upon library installation. So it seems that the IDE does not play well with dropbox sync client (as of now v113.4.507).
So I tried it also with Nextcloud client v3.1.1 which did not show the directory name issue.
Also megasync client v4.3.8 worked well.

After all the testing I switched back to dropbox managed folder and did a run with Sysinternals Process Monitor monitoring filesystem operations on a libraries test folder. Find it attached with an image of the Process Monitor filter settings to this post. I suppose the interesting two lines are the ones with Result equal to SHARING VIOLATION and INVALID DEVICE REQUEST. So what to do with this information? Maybe you have a contact to direct this to. Maybe a retry in the IDE could fix the race condition(?) with concurrent access by IDE and external programs.

[pert]

Thanks so much for taking the time to investigate this and share your findings @alex_73! Having confirmation that this can cause the issue will enable me to suggest it as something for users to check without being afraid that I'm sending them off on a wild goose chase.

As for bringing this to the attention of the people who can fix it, your issue in the arduino/Arduino GitHub repository's issue tracker is a reasonablethe best place. Unfortunately, the Arduino IDE very popular piece of software maintained by only a couple developers so things to move slowly, but with patience I have seen many of the bug reports and feature requests I've made over the years resolved.

Arduino has been on a campaign for years now to move all the underlying functionality out of the Arduino IDE's Java code base and into a command line tool (which was originally arduino-builder, but later became Arduino CLI), eventually making the Java code base purely for the GUI components of the IDE. Arduino CLI is used by Arduino IDE, Arduino Web Editor, Arduino Pro IDE, and perhaps other 3rd party development software as time goes on. And of course it can be used directly. So this means that a fix or improvement made in Arduino CLI will eventually propagate to all the other Arduino development software. Development in the Arduino CLI is more active than in the Arduino IDE repository (though of course work done on Arduino CLI is also work on the IDE in the end).

I notice that when I try to update a library in Arduino CLI with a library file or folder open it does fail with a more helpful error message:

Code: [Select]

$ ./arduino-cli lib install Ethernet
Ethernet depends on Ethernet@2.0.0
Downloading Ethernet@2.0.0...
Ethernet@2.0.0 already downloaded
Installing Ethernet@2.0.0...
Replacing Ethernet@1.0.0 with Ethernet@2.0.0...
Error installing name:"Ethernet" versionRequired:"2.0.0" versionInstalled:"1.0.0": moving extracted archive to destination dir: rename C:\Users\per\Documents\Arduino\libraries\package-727220067\Ethernet-2.0.0 C:\Users\per\Documents\Arduino\libraries\Ethernet: Access is denied.

I see that the original library ends up removed after this occurs and I'm not sure that's the most ideal behavior, but the error message makes it clear that something went wrong and will at least give a clue as to what it was that happened. In some cases you could probably get a successful install on the second attempt.

So a feature request for a retry might actually be most appropriate in the Arduino CLI issue tracker.

[alex_73]

Thanks so much for taking the time to investigate this and share your findings @alex_73!

You are welcome .

Development in the Arduino CLI is more active than in the Arduino IDE repository (though of course work done on Arduino CLI is also work on the IDE in the end).

As much as it make sense, this reads for me that arduino devs will rather have arduino-cli integrated to the UI than fixing (minor/rare?) issues in the IDE. I mean how many ppl have their arduino libraries replicated to dropbox like I do...

I notice that when I try to update a library in Arduino CLI with a library file or folder open it does fail with a more helpful error message:

I installed arduino-cli v0.14 today and got it working to the point of installing libraries to the dropbox managed libraries folder. Interestingly enough, directories here are created as expected. No temporary directory names to be seen. No error messages. Updates of existing libs also worked flawlessly as far as I could test. So "my" issue really relates only to the IDE. Sigh.
As you found the arduino-cli issue, will you report it on github? I'll create an issue for the IDE even though chances are low that it will get fixed.
Created: https://github.com/arduino/Arduino/issues/11203

[alex_73]

I need to correct myself. I just got errors also with arduino-cli in dropbox and stale directories named package-\d+:

Code: [Select]


d:\CloudStorage\Dropbox\Work\Arduino\libraries>arduino-cli lib install "Arduino Cloud Provider Examples"@1.0.0
Arduino Cloud Provider Examples@1.0.0 depends on Arduino Cloud Provider Examples@1.0.0
Downloading Arduino Cloud Provider Examples@1.0.0...
Arduino Cloud Provider Examples@1.0.0 already downloaded
Installing Arduino Cloud Provider Examples@1.0.0...
Installed Arduino Cloud Provider Examples@1.0.0

d:\CloudStorage\Dropbox\Work\Arduino\libraries>arduino-cli lib install "Arduino Cloud Provider Examples"@1.1.0
Arduino Cloud Provider Examples@1.1.0 depends on Arduino Cloud Provider Examples@1.1.0
Downloading Arduino Cloud Provider Examples@1.1.0...
Arduino Cloud Provider Examples@1.1.0 already downloaded
Installing Arduino Cloud Provider Examples@1.1.0...
Replacing Arduino_Cloud_Provider_Examples@1.0.0 with Arduino Cloud Provider Examples@1.1.0...
Installed Arduino Cloud Provider Examples@1.1.0

d:\CloudStorage\Dropbox\Work\Arduino\libraries>arduino-cli lib install "Arduino Cloud Provider Examples"@1.2.0
Arduino Cloud Provider Examples@1.2.0 depends on Arduino Cloud Provider Examples@1.2.0
Downloading Arduino Cloud Provider Examples@1.2.0...
Arduino Cloud Provider Examples@1.2.0 already downloaded
Installing Arduino Cloud Provider Examples@1.2.0...
Replacing Arduino_Cloud_Provider_Examples@1.1.0 with Arduino Cloud Provider Examples@1.2.0...
Error installing name:"Arduino Cloud Provider Examples" versionRequired:"1.2.0": moving extracted archive to destination dir: rename d:\CloudStorage\Dropbox\Work\Arduino\libraries\package-120976539\Arduino_Cloud_Provider_Examples-1.2.0 d:\CloudStorage\Dropbox\Work\Arduino\libraries\Arduino_Cloud_Provider_Examples: Access is denied.
So it seems dropbox is not a good option for replicating arduino code.