Sharing source files across projects or sketches

Projects Programming
1

Hi I have a few sketches that are going to share some source code. For example, I have a cpp file for string manipulations to construct a parser. I am writing code for new sensor boards with ADC, DAC and TDC chips, so there will be a cpp for each of those, and etc.

Now, admittedly these are a lot like libraries. In fact they're libraries. But I do not want to put these in the arduino tree at this time for a few reasons including that (a) they are not ready for release, (b) they are specific to this project and not general use, (c) I want to avoid name conflicts with the rest of the arduino universe and (d) I do not want the overhead of having to remember to copy and set them up in a new tree when the arduino software is updated or replaced or when I setup a new host computer to work on.

So for all of the above, "put them in the arduino tree" is not what I am looking for at the moment. Since some of these are about interface chips (ADC's and DAC's and TDC's), I probably will contribute them in future, when I learn how and am also satisfied with the code. For now, I need to keep them separate and share them across a few sketches.

I envision two possible solutions, maybe there are more.

a) How do I use code from a directory that is outside of the sketch directory and not in the arduino library tree?

B) Is it possible to have a private library tree? If so, please be incredibly specific about how to setup the directories for the headers and source files.

Thank you

2

I do it like this: [#include "/home/gil/Gil_Lib/Gil_Net.h" // define wifi credentials] without the braces. I use the full path, this is a linux not a windoze box but it should work by using the quotation marks and the .

1 Like
3

Hi @DrN. The way to share code between Arduino projects is libraries. So you should install the files as a library by putting them in a folder under the libraries subfolder of your sketchbook folder.

If you don't know the location of your sketchbook folder, select File > Preferences... (or Arduino IDE > Settings... for macOS users) from the Arduino IDE menus and note the path shown in the "Sketchbook location" preference.

What do you mean by the "arduino tree"? Do you mean installing them as libraries in your sketchbook as I recommend?

I'll address your excuses for not wanting to do that based on the assumption that is what you mean:

That is irrelevant. There is no need to release the code. I'm not talking about submitting the libraries to the Arduino Library Manager.

You only need to manually copy the files to a folder under the libraries subfolder of the sketchbook folder.

That is not a strong argument against installing them as a library. It is very common for us to install a library that will only be used for a single project.

If you follow best practices when writing your code that won't be a serious problem. Just make sure to use a distinctive name for the library and then use that name in the header files and classes/namespaces. There won't be any problem with name collisions of even commonly named objects within that scope.

This will be important once the time comes for you to publish the code anyway.

That won't be necessary. Updating or replacing the Arduino software doesn't affect your sketchbook contents.

You would need to copy the files to the new computer regardless.

When it comes to code in .cpp files, there is no reasonable way to do it if you are using Arduino IDE. However, if you are using the official command line tool provided by Arduino: Arduino CLI, then it is possible to specify additional paths to individual libraries via the --library flag, or the path of an additional folder containing a collection of libraries via the --libraries flag of the arduino-cli compile command.

You can learn how to use that command from the reference here:

https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/

The easiest way will be to use a dedicated sketchbook folder for your project, which will contain the libraries for that project. Although not required, it will be convenient to also store the project sketches in that folder.

You will configure the Arduino IDE preferences to use that sketchbook location when working on that project. When you want to work on unrelated projects, you would change the sketchbook location to your standard sketchbook.

If you are using Arduino IDE, you can do that through the Preferences dialog. It is also possible to use a script to change the sketchbook location preference if you prefer to make the process less manual. If you are interested in that, we can provide instructions but would need to know which Arduino development software you are using (e.g., Arduino IDE 2.x, Arduino IDE 1.x, Arduino CLI)

I'll provide an example folder structure. Assuming you have two libraries: "Foo" (which is made up of files Foo.h and Foo.cpp) and "Bar" (which is made up of files Bar.h and Bar.cpp), the structure would look like this:

<sketchbook location>/
├── libraries/
│   ├── Bar/
│   │   ├── Bar.cpp
│   │   └── Bar.h
│   ├── Foo/
│   │   ├── Foo.cpp
│   │   └── Foo.h
│   ...
...

(where <sketchbook location> is the path you set in the Arduino IDE "Sketchbook location" preference)

This will only give access to the code in the header file. Unlike with an #include directive for a header file in a normally installed library, this will not result in the compilation of source files in /home/gil/Gil_Lib/.

1 Like
4

Any non-trivial project should be under source control. If you use Git for example, it has a submodules feature.

Your sketch should then have the following structure

  • mysketch/
    • mysketch.ino
    • arduino_secrets.h
    • src/
      • some_code.cpp
      • mysketch_only_stuff/
        • more_constants_or_whatever.h
      • submodule1/
        • submodule1.h
        • submodule1.cpp
      • submodule2/
        • submodule2.h
        • abc/
          • abc.h
          • abc.cpp

Each submodule is its own repository, and different sketches can share that code, at exactly the same commit, or at different commits if needed.

With a single git pull --recurse-submodules you get everything (almost) every time, without having to do anything "outside the tree". New collaborators just need access to the individual git repos.

1 Like
5

I am going to wind up apologizing for this, I am sure, but I think it really needs to be said. So I will do my best to filter and be positive. For your side, please try to listen and consider that I am not the first one to ask this, not by far, and that even if perhaps i am still finding my way around arduino, I have been programming on the metal just about as long as the metal has existed to program on.

Being forced/compelled to put the code in the library tree means every arduino sketch that I work on sees it. That imposes a constraint on some part of the name space (or filename space) for each of those projects. That constraint does not exist for any other reason except for the lack of a way to specify an additional tree to search for headers and code. In other words that is an entirely artificial constraint. When I find an artificial constraint in in my own work, I usually take that as a sign that I should spend some time thinking more deeply about the design.

The answer, "conform to naming conventions", makes assumptions or imposes another set of artificial limitations on how people are working and the kinds of things they work on.

All of that so far is maybe abstract, it is mathematically a solid point, but lets see if I can come up with something close to a specific example.

Suppose I have three sketches that all use a function megillicuty(), it could be a debugging function for example and it is specific to some hardware that all three sketches are working with.

Now lets say I have another three sketches that have the same structure, but different hardware, so I rewrite the shared code a bit for those, I still have the same names, because it is my private work and it is easier for me to keep track of it in private libraries rather than being forced to share the libraries across two different projects and accidentally calling something from the wrong library over an editing error.

Now in the larger view, I should not be forced to call all of the functions in one library according to the convention megillicuty_for projects_abc() and the other megillicuty_for_projects_def() only because an ide or compiler chain is incapable of accepting an -include and -library somewhere in its build environnent. Rather, the correct approach is to fix the ide.

And from a still larger point of view, It is a perfectly reasonable expectation to have a set of libraries for projects a,b and c, and another for d,e and f, and, be able to compile and link without and IDE trying to tell me how to live my life as a programmer.

So, now with apologies if i crossed over to flaming, the answer that i should addess this by conforming to naming conventions, really misses the point.

Please think about how to add something along the lines of include and library to the IDE.

Thank you

6

I'm using linux

7

I have listened. I'm here to offer solutions for working with the existing capabilities of the tools, and that is exactly what I have done. If you were expecting me to magically materialize capabilities that don't exist then I'm afraid I can't help you.

Thanks for your suggestion. However we are satisfied with the current system. 99.9% of the many thousands of users of this IDE have been satisfied with it as well, over the course of nearly two decades of great popularity. So the requested feature will not be added.

We already provide the requested feature in Arduino CLI, which is targeted to the requirements of advanced users like yourself.

If you aren't satisfied with the capabilities of the free open source tools offered by Arduino, feel free to use one of the excellent alternative tools.

8

I appreciate the suggestion use github. I thought about it and decided I am not doing that for this project at this stage, for whatever reasons. (It would be off topic for this question).

Aside, regarding "should", I am happen to like github, it is pretty neat and especially for collaboration, but there are other ways to manage large code projects that work well also.

9

You have probably noticed that Arduino, or at least the paradigm of an mcu or arm processor on a small format board, is expanding into product deployments and even high performance applications that leverage the peripheral connectivity. That is a testimonial to what you have accomplished by establishing a strong foothold in the diy world. But nothing stands still.

The CLI sounds like a great suggestion. I imagine it could be more convenient for deployment and compatible with the sort of make file one usually includes in a github repo. Thank you for that suggestion, I will look into it.

Does the CLI work with Teensy? (I should search first before asking, but as long as you're here...)

10

Yes it does. Arduino CLI actually provides all the non-GUI functionality for Arduino IDE under the hood. So anything you can do in Arduino IDE can also be done via Arduino CLI, plus much more!

11

That is correct. I have quit using libraries for the most part as I find later code will not compile or work or both because of library changes. With the low cost of file storage and the large drives it does not impact much but the code complies years later.

12

Great that sounds exciting, I've also been looking for conditional compiles for a long time.

14

Yes, I realize git is separate from github. so far, git by itself doesn't do much for me.

And, on the subject of ide's, my preferred environment is emacs and make. it is robust and there is rarely anything hidden. So the cli support is what seems most interesting.

Hearing that the arduino ide is for noobs is news. Would you say that any of these other ides have a large growing user base compared to the arduino ide and at least equally reliable support?

One of the hard lessons that few would want to repeat, is investing a lot of code in a cushy environment and then having it go away.

16

So, I looked into the CLI. It looks nice, but I still have some questions.

How would I specify where to look for extra .c, .cpp and .h files?

Aside, I have about ten boards that I need to code for, all of which I want to post to git-hub, and there is a lot of shared code across the collection.

I envision one directory for the shared code and individual directories for the sketch for each board.

Thank you

17

There are two options. You can use whichever you prefer:

Use the --library flag

Pass the path to a folder the root of which contains .c, .cpp and .h files to the arduino-cli compile command via the --library flag.

For example, if you have a folder structure like this:

~/some-project/
├── some-libraries/
│   ├── ADC/
│   │   ├── ADC.cpp
│   │   └── ADC.h
│   ├── DAC /
│   │   ├── DAC .cpp
│   │   └── DAC .h
│   ...
...

Then you can run this command:

arduino-cli compile --library ~/some-project/some-libraries/ADC [...]

With that flag added to the command, having this line in the sketch program:

#include <ADC.h>

Will result in the discovery of the library at ~/some-project/some-libraries/ADC. The path of that library will be passed to the compilation commands via a -I flag and the source files in the folder will be compiled.

Use the --libraries flag

Pass the path to a folder which contains subfolders the roots of which contain .c, .cpp and .h files to the arduino-cli compile command via the --libraries flag.

For example, if you have a folder structure like this:

~/some-project/
├── some-libraries/
│   ├── ADC/
│   │   ├── ADC.cpp
│   │   └── ADC.h
│   ├── DAC /
│   │   ├── DAC .cpp
│   │   └── DAC .h
│   ...
...

Then you can run this command:

arduino-cli compile --libraries ~/some-project/some-libraries [...]

With that flag added to the command, having this line in the sketch program:

#include <ADC.h>

Will result in the discovery of the library at ~/some-project/some-libraries/ADC. The path of that library will be passed to the compilation commands via a -I flag and the source files in the folder will be compiled.

You can learn about these flags and other information about the arduino-cli compile command from the documentation here:

https://arduino.github.io/arduino-cli/latest/commands/arduino-cli_compile/

For the sake of simplicity, I have used the minimal "flat" library layout structure in my examples. It is also possible to organize the code files of more complex libraries into subfolders if you like. You can learn about doing that from the Arduino Library Specification:

https://arduino.github.io/arduino-cli/latest/library-specification/#layout-of-folders-and-files

1 Like
19

I see, very nice.

I am contemplating something like the following.

~/some-project/
|---mySharedCodes/
| |-functions.cpp
| |-functions.h
| |-otherfunctions.c
| |-otherfunctions.h
| |-etc
|---boardA/
| |-boardA.ino
|---boardB/
| |-boardB.ino
|---boardC/
| |-boardC.ino
|---etc/

Presumably the switch is --library ../mySharedCodes ? (in other words, does it accept a relative directory path?)

Thank you

20

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.