Arduino IDE can't find local (custom) libraries in relative path

Projects Programming
1

I'm trying to restructure my project and get everything in git, especially the custom headers I share between my Arduino & RPi code. I moved all the code into a single project directory with three subdirs: "server", "client", and "includes" and copied my sketch into the "client" dir. I then changed the include for my header from e.g. '#include <myfile.h>' to '#include "../includes/myfile.h"' and while this is perfectly valid C/C++, the arduino compiler doesn't like it and complains about not being able to find the file.

myproject:8:10: fatal error: ../includes/myfile.h: No such file or directory

I'm not sure why this is happening. It's always been my understanding that relative paths in double-quoted include directories are interpreted as being relative to the source file in question, not e.g. the current working directory or anything like that. I use the same include in the c++ RPi project (in the "server" directory) without issue.

Full disclosure: I'm trying to switch to VS Code since the Arduino IDE is very basic/immature, so I'm looking for a solution that doesn't involve changing IDE settings -- since those probably won't carry over. The project runs into the same error with the Arduino IDE and VS Code, so I think it has something to do with the avr-gcc flags, so maybe I can fix it somehow with a change to platform.txt?

2

The Arduino IDE does not work with relative paths. You need to provide the full path name if the library is not in the same folder as your .ino file. This is because the IDE copies the files to a different folder before compiling and the relative references don't work from that different folder.

And, I agree, it's a real PITA. I created a Python program to compile and upload using the command line IDE and the Python program converts relative paths to absolute paths before calling the IDE

...R

3

Oh wow, that really sucks. Do you know of any documentation or discussions that explain why this is the case, and any other "gotchas" with the environment?

I tried setting up symlinks in the directory with the INO file and that didn't work either, so I'll have to follow your example and come up with something similar for VS Code.

I think I'm going to open an issue about this, if there isn't one already.

4

alzee:
Do you know of any documentation or discussions that explain why this is the case, and any other "gotchas" with the environment?

The Arduino IDE is designed to be easy for a complete newbie to use and my guess is that the way it is implemented "seemed like a good idea at the time" - after all, who would expect newbies to understand relative paths :slight_smile:

I don't use the IDE for editing my Arduino programs - I use the Geany editor for all my programming.

...R

5

The Arduino IDE likes for custom libraries to be stored in a folder named "src" in the sketch folder. Then you can include the file with #include "src/libraryfolder/libraryheader.h". May work if you move your folder into src then include the file with "src/includes/myfile.h"

1 Like
6

Robin2:
The Arduino IDE is designed to be easy for a complete newbie to use and my guess is that the way it is implemented "seemed like a good idea at the time" - after all, who would expect newbies to understand relative paths :slight_smile:

Yeah I get that and have heard this sort of argument regarding the Arduino IDE already, my retort is always the same though: Supporting "complex" features doesn't have to detract from the simplicity of the overall system.

I don't use the IDE for editing my Arduino programs - I use the Geany editor for all my programming.

...R

I actually ran into this trying to get this working with VS Code, as I'm trying to get away from the Arduino IDE and all it's limitations. I wanted to see if it would work in the official IDE before making a post. The error is identical in both so I was pegging it to the avr-gcc compiler and not the IDE specifically.

Right now it feels like I might be just moving to WinAVR etc, but the jump to there from working Arduino code seems like it could be challenging.

7

Robin2:
You need to provide the full path name if the library is not in the same folder as your .ino file.

There is a huge caveat to this. It's true that you can provide the full path name to a header file, but this will not cause the source files in that path to be compiled, as is done when you only provide a header file name in the #include directive without a path. So as far as saying "library", that is only correct when it's a header-only library, which are relatively rare.

alzee:
Do you know of any documentation or discussions that explain why this is the case

It's here:
https://arduino.github.io/arduino-cli/latest/sketch-build-process/#compilation
The reason for compiling the sketch in a temporary directory rather than in place is that, to make things easier for beginners, a couple things are automatically done to the .ino files of a sketch to make them a valid C++ file before compiling, as described here. So the code that is compiled is different from the code in the original folder. If you enable File > Preferences > Show verbose output during: compilation and then check the contents of the black console pane at the bottom of the Arduino IDE window after compiling, you can see the location of the temporary build folder and check out the contents.

alzee:
and any other "gotchas" with the environment?

The Arduino sketch specification might also provide some useful information.

alzee:
I think I'm going to open an issue about this, if there isn't one already.

There definitely is one, but I don't have time to search it out right now. There will definitely be some interesting discussion there, so I recommend you to check it out and comment if you have anything original and productive to add. It will likely be here, as that was the relevant issue tracker up until 5 years ago:

For the last 5 years, the more relevant location has been here:

and the current relevant location is here:

It would be great if you'd post the link here once you find it.

8

OK, I see you're commenting on this issue:

That reminds me that I meant to point out this feature:
https://arduino.github.io/arduino-cli/latest/sketch-specification/#src-subfolder
This would allow you to put the shared header files in a subfolder of your sketch folder:

MySketch
|_ MySketch.ino
|_ src
   |_ includes
      |_ myfile.h

Then you can do this in your sketch:

#include "src/include/myfile.h"
9

pert:
There is a huge caveat to this. It's true that you can provide the full path name to a header file, but this will not cause the source files in that path to be compiled, as is done when you only provide a header file name in the #include directive without a path. So as far as saying "library", that is only correct when it's a header-only library, which are relatively rare.

I know this was to Robin, but I kind of expected this myself, which is another reason I want to move away from the Arduino IDE. The lack of traditional c/c++ multiple-source file support struck me early on when I started my second or third sketch and I couldn't figure out how to add another file to it in the IDE.

The reason for compiling the sketch in a temporary directory rather than in place is that, to make things easier for beginners, a couple things are automatically done to the .ino files of a sketch to make them a valid C++ file before compiling, as described here. So the code that is compiled is different from the code in the original folder. If you enable File > Preferences > Show verbose output during: compilation and then check the contents of the black console pane at the bottom of the Arduino IDE window after compiling, you can see the location of the temporary build folder and check out the contents.

Cool thanks, I may look into that just to satisfy my curiosity.

There definitely is one, but I don't have time to search it out right now. There will definitely be some interesting discussion there, so I recommend you to check it out and comment if you have anything original and productive to add. It will likely be here, as that was the relevant issue tracker up until 5 years ago:
Issues · arduino/Arduino · GitHub
For the last 5 years, the more relevant location has been here:
Issues · arduino/arduino-builder · GitHub
and the current relevant location is here:
Issues · arduino/arduino-cli · GitHub
It would be great if you'd post the link here once you find it.

I'll go searching, thanks, and I'll certainly get back here with the link if I find it. I already posted something on the wrong PR for something I found and didn't read closely, as you have already discovered. Before that though, I need to go relax for a bit.

Right now I'm thinking what I'll do, since the symlinks don't work, is just put a pre-build step into VS Code to copy the relevant files from their revision controlled directory to the sketch directory (or src subdir if needed) before kicking off the build, to see if that gets me where I need to be.

Up until now I've just been manually copying the shared files around so I can use a single set between Arduino and the RPi, but that's starting to get old.

10

pert:
OK, I see you're commenting on this issue:
Support bundling libraries with a sketch · Issue #1255 · arduino/arduino-cli · GitHub

That reminds me that I meant to point out this feature:
Redirecting
This would allow you to put the shared header files in a subfolder of your sketch folder:

MySketch
|_ MySketch.ino
|_ src
|_ includes
|_ myfile.h

Then you can do this in your sketch:

#include "src/include/myfile.h"

I'm going to try this with the symlinks and see what happens. Thanks.

11

alzee:
I'm going to try this with the symlinks and see what happens. Thanks.

Unfortunately this doesn't seem to work either. Instead of complaining about not being able to find the file, it complains that the file itself is invalid with an error like:

sketch\src/myheader.h:1:1: error: expected unqualified-id before '.' token

 ../../../includes/myheader.h

So it's finding the file and the #include directive is working ok, but the symlinks aren't being parsed when the files are copied I guess.

12

You're right Juraj.

One thing to consider though: the Arduino IDE has a nasty behavior of discarding everything from the sketch except what is provided for in the specification when you do a "save as".

It's true that, even though the src subfolder is now in the specification, it also discards that folder, but that is considered a bug, whereas the discarding of arbitrary folders is not necessarily considered a bug (though I personally think it should be). So I would say that using the src subfolder is more of a best practice than using an arbitrary folder.

On the other hand, by this same logic you could argue that putting header files under the data folder is superior to putting them under src, since the data folder actually is preserved by the Arduino IDE. However, this is somewhat contrary to the intended use of the data folder and you could probably consider the fact that the data folder is being copied to the temporary build folder to be a bug, especially since that is the location where relatively large files such as PDF, images, CAD files, etc. may be stored.

13

pert:
You're right Juraj.

sorry, I realized that the builder doesn't copy other folders to build folder and deleted the comment. if you can undo the deletion I would strike the text

14

pert:
There is a huge caveat to this. It's true that you can provide the full path name to a header file, but this will not cause the source files in that path to be compiled,

Good point. I only ever use header files.

pert:
So I would say that using the src subfolder is more of a best practice than using an arbitrary folder.

The problem with the src subfolder is that it cannot be shared among multiple Arduino projects - for example if you want a file of common connection data for a wireless TX program and the corresponding wireless RX program.

...R

15

Juraj:
sorry

No worries. It really doesn't matter.

Juraj:
the builder doesn't copy other folders to build folder

I expected that to be the behavior, but my quick tests indicate that it does copy everything. However, I didn't check comprehensively. Did you find that it was not copied? If so, which IDE/whatever version and which OS?

I'm just interested in learning more about the behavior of the various versions of the various official development tools.

When I wrote the Arduino sketch specification, it ended up being kind of messy because the Arduino IDE and Arduino Web Editor have different behavior, some of which is even conflicting. Then there are some differences with the Arduino CLI as well. It made me think of how it's valuable to have a well defined specification that is followed consistently by all the development tools as much as possible. This way, sketch authors and users have a pretty good idea of what should be supported. If they find that something in the specification is not working, that's a bug that should be reported and fixed. If they are doing something that is not specifically defined in the specification which just happens to work by chance, they shouldn't complain when it later stops working. Without a formal specification, it's all just guesswork.

1 Like
16

Robin2:
The problem with the src subfolder is that it cannot be shared among multiple Arduino projects - for example if you want a file of common connection data for a wireless TX program and the corresponding wireless RX program.

Exactly. This is why I normally recommend that people just install libraries the usual way. However, there are a few specific use cases where bundled libraries are very useful. I have used them when I wanted to share sketches with a friend who was not very experienced with Arduino. I wanted to be able to provide a self-contained package that didn't have any external library dependencies as I knew they would find the need to install a bunch of libraries to be intimidating.

It could also be useful if you want to pin the version of the library used by the sketch, so that it's unaffected by updates to the sketchbook installed copy of the library.

You could use a dedicated sketchbook, or even a dedicated portable installation of the Arduino IDE for this purpose, but neither is as convenient for the user as just being able to use the sketch as usual.

17

pert:
Exactly. This is why I normally recommend that people just install libraries the usual way.

Perhaps. But you wouldn't want to put a file that has two or three lines such as the radio channel and communication speed into the library folder. Apart from anything else it would be a real PITA if (for example) you want to change the channel.

Also, I am not a great fan of a centralised library where there is a risk of an updated library for one project breaking an older project that was using the earlier version of the library. Python has the nice feature of virtual environments in which it installs the downloaded library files into a folder that you specify - so the libraries can remain specific to a project.

...R

18

Robin2:
it would be a real PITA if (for example) you want to change the channel.

Why is it a PITA? It's just as easy to edit files located in the libraries folder as anywhere else, right?

Robin2:
Python has the nice feature of virtual environments in which it installs the downloaded library files into a folder that you specify - so the libraries can remain specific to a project.

The Arduino equivalent of that would be to have a dedicated sketchbook for the sketch.

There is some discussion about adding an "environments" capability:

That's in the Arduino CLI repo, but any functionality added to Arduino CLI becomes available to the other tools if they chose to use it. It was considered an important enough idea to be made one of the two "pinned" issues out of nearly 100 in that repository.

19

pert:
Why is it a PITA? It's just as easy to edit files located in the libraries folder as anywhere else, right?

You obviously have a much greater tolerance for pain :slight_smile:

...R

20

Robin2:
The problem with the src subfolder is that it cannot be shared among multiple Arduino projects - for example if you want a file of common connection data for a wireless TX program and the corresponding wireless RX program.

This is exactly my use case, and what set me down this path. I have a couple of header-only libraries that define data structures and such that I need to share between all of the projects. This is pretty easy to do (and share with others) with a traditional C/C++ project where everything is kept together shared via e.g. github.