Why can't the .ino file remember which libraries (and linbrary versions) were used by that sketch?

Development Suggestions for the Arduino Project
1

(maybe there is a better workflow that I should be using to avoid this problem, I am open to suggestions).

So lets say I create a sketch and it runs great on the target hardware. I save the .ino file for posterity on my NAS, or on github or wherever.

The next time I want to use that sketch, I might be on a new computer or at a different location. Unless I had meticulously documented which libraries I had had installed (and which specific versions of those libraries) I might not even be able to get the sketch to compile anymore.

So when I save my sketchbook, why can't the .ino file include information about what libraries are used, and maybe even what board and what board options were used?

2

Hi @muzzlevelocity. The .ino file contains Arduino programming language/C++ code. It would not make sense to put data other than that code in the .ino file

The correct way to do something like this is to put the information in a dedicated metadata file in an appropriate language (e.g., YAML, JSON). And in fact Arduino has done exactly this already with the Arduino sketch project file:

https://arduino.github.io/arduino-cli/latest/sketch-project-file/

You can use the official Arduino CLI command line tool to manage the dependencies of your sketch projects with a project file:

https://arduino.github.io/arduino-cli/latest/

3

"Clone" the library directory/folder with the .INO file. If you use the Arduino IDE, the local files will be tabs behind the .INO file.

4

Use git submodules and clone libraries instead of just downloading them. Then whenever you check out your old code and its submodules, you'll still be working with the old versions even if you've updated a thousand times since then.

5

An easy way is to add the meta info as comments in the header of the ino file

e.g

//
//    FILE: temperature_scope_test.ino
//  AUTHOR: Rob Tillaart
// PURPOSE: test sketch for scope
//     URL: https://github.com/RobTillaart/ TODO
//
// Uses
// - https://github.com/adafruit/Adafruit_AHTX0 version 2.0.5
// - https://github.com/RobTillaart/MCP3424 version 0.1.1
// - 
// Compiles with IDE 1.8.19
// - file size 6714 
//
2 Likes
8

just a hint:
always note beside the include, how the library is named and which version was used with the last compilation.

Also note the board core version in your header comments.

I advice against a local copies of the library in the sketch folder, because you will end up with local duplicates of libraries - and these copies will not be updated anymore. Furthermore you end up with different versions of libraries for your projects. So you might wonder why it is working in one sketch but not anymore in the other sketch due the version differences.

9

My approach to keep libraries in the sketch folder and include them with "" instead of <> is a valid way to ensure your code uses the specific versions of libraries you intend. However, there are some best practices and additional details to consider that can make this approach more robust:

Refined Approach

  1. Create a libraries Folder in the Sketch Directory:
  • In your sketch folder, create a subfolder named libraries to organize all your custom or older library versions. This will keep things tidy and ensure the compiler finds the libraries you intend to use.
  1. Include the Libraries Correctly:
  • When including libraries, use the relative path from the sketch folder. For example:
#include "libraries/MyLibrary/MyLibrary.h"
  • This ensures that even if there are other libraries with the same name in the global library folder, the compiler will prioritize the local version.
  1. Avoid Library Conflicts:
  • Remove or rename the same library from the global library directory (usually in Documents/Arduino/libraries) to avoid conflicts and ensure the compiler does not pick up the wrong version.
  1. Version Control:
  • Use version control (like Git) for your projects. This way, if a library update breaks something, you can easily roll back to a previous version of your code and the libraries.
  1. Backup Libraries:
  • If possible, keep a backup of the libraries with your project. You can even create a backup folder within your libraries folder to store older versions as a precaution.
  1. Documentation:
  • Document the version of the library you are using within your code or in a separate README file. This helps in troubleshooting and ensures anyone working with your code knows the dependencies.

Example Folder Structure

scss

Copy code

MyProject/
├── MyProject.ino
├── libraries/
│   ├── MyLibrary/
│   │   ├── MyLibrary.h
│   │   ├── MyLibrary.cpp
│   │   └── ... (other files)
│   └── AnotherLibrary/
│       ├── AnotherLibrary.h
│       ├── AnotherLibrary.cpp
│       └── ... (other files)
└── README.md

Example of an #include Directive

#include "libraries/MyLibrary/MyLibrary.h"
#include "libraries/AnotherLibrary/AnotherLibrary.h"

Additional Tips

  • Compile Path Issues: Make sure your Arduino IDE settings are configured to recognize local libraries first. Sometimes the IDE might still prioritize the global library path.
  • Testing: Test your code periodically with the latest versions of the libraries to identify and fix potential issues in a controlled environment.

By implementing these strategies, you'll have a more robust setup that reduces the risk of library updates breaking your old code while keeping everything organized and maintainable.

Hopefully this helps.

10

@gilshultz these are detritus from the UI on the ChatGPT code blocks. Make sure to strip them out of the answer after you paste it to the forum.

11

That does not work (and to my knowledge has never worked). The IDE will not compile .cpp files in random directories.

There is a dedicated directory for what you describe; it's called src, you just need to create it.

That would explain why it does not work :rofl:

1 Like
12

Something that might contribute to the confusion is that the #include directives for header files under a subfolder with name other than src will work. However, the source files under that subfolder will not be compiled.

For example, if you have this:

libraries/MyLibrary/MyLibrary.h:

void foo() {}  // Inline function
void bar();  // Declaration

libraries/MyLibrary/MyLibrary.cpp:

void bar() {}

Then this sketch will compile:

#include "libraries/MyLibrary/MyLibrary.h"
void setup() {
  foo();
}
void loop() {}

but this sketch will not:

#include "libraries/MyLibrary/MyLibrary.h"
void setup() {
  bar();
}
void loop() {}

C:\Users\per\AppData\Local\Temp\ccpgwTLf.ltrans0.ltrans.o: In function `setup':
C:\Users\per\Documents\Arduino\MyProject/MyProject.ino:3: undefined reference to `bar()'
collect2.exe: error: ld returned 1 exit status
1 Like
13

Here is how I have it in many sketches.

//#include <UnixTime.h>  // Origional one
#include "/home/gil/Gil_Lib/UnixTime/UnixTime.h"

This has been working for me for several years. If it is in the sketch directory I do it this way

#include "UnixTime.h"

Thanks, I used that to format the answer.

14

It will work if all the referenced code is in the header file, but it won't work if you reference objects defined in a source file at that location (e.g., /home/gil/Gil_Lib/UnixTime/UnixTime.cpp).

15

Thanks for the update. I believe I understand what you're saying. I’ve placed the .cpp files and other related files in the sketch directory. Sometimes these files include references to additional libraries, so I usually copy those over as well and adjust the include statements accordingly. This approach uses more disk space, but it’s easier than trying to track down a library that may have changed something.

I try to avoid using external libraries whenever possible and prefer to write the code myself or extract only the necessary parts from a library.

16

Just to be pedantic, I'll point out that various "tools" have been capable of parsing environment meta-data from the comments of source files for ... decades now.
For example: File and directory local settings - Configuration - Doom Emacs Discourse

It's not like the Arduino IDE doesn't already do significant parsing of your source code before actual compilation.

A trivial but perhaps useful example, the IDE "include library" command could comment the #include that it inserts with the version of that library - it already HAS that data...

17

Just because it is possible doesn't make it a good idea.

I don't know why the Arduino community always has this idea of polluting the sketch this way. I guess it comes down to lack of familiarity with the established dependency management systems, all of which use dedicated machine readable files for the dependencies data.

Yes, and this was very difficult to implement and is still imperfect.

Compare that to reading a file in a standard machine readable data format, which can be done 100% reliably using a standard package with a few lines of code.

It makes sense to parse the source code for the source code because that is the only way. It doesn't make any sense to parse the source code to look for data in proprietary meta comments.

And do you want it to cram in all the information about all the transitive dependencies in that comment as well?

18

It probably doesn't need that, since if you retrieve the old library version, it can do that itself (except that I don't think that libraries currently include version numbers for transitive dependencies.)

Hmm. It might be nice to save the output of the whole "libraries detection phase" in a log file somewhere... (but I guess arduino-cli could do that?)

19

They can:

https://arduino.github.io/arduino-cli/latest/library-specification/#version-constraints

20

How many third party libraries actually do? My bet would be not many.