My Libraries & Files are all over the Place

Back when I started learning arduino, I decided that microsoft was not going to tell me where to keep my files. and so I set up at least two locations other than under users. Now I am older and more tired, and trying to figure out which library is where, and which of the various folders the sketches are in is just too much.

I want to move everything to where it is supposed to be so that Arduino finds the file/library/sketch when it is looking for it. But I don't know where that is.

Will someone tell me where to put everything, and how to tell the IDE to look there ?

  • Decide on a convenient location on your computer to store your sketches and libraries.
  • Set that location as your sketchbook folder in the Arduino IDE at File > Preferences > Sketchbook location
  • Put all the sketches inside the sketchbook folder. You can use any subfolders you like to organize them but don't put any sketches in the libraries subfolder of the sketchbook folder. The only sketches that should be in the libraries folder are the sketches found in the examples subfolders of the individual library folders. You will be able to access all your sketches from the File > Sketchbook menu of the Arduino IDE.
  • Put all your 3rd part and custom libraries in the libraries subfolder of the sketchbook folder. The individual library folders should be directly under the {sketchbook}/libraries folder, not in subfolders.

I also shun the stupid "My Documents", etc. folders that Windows wants you to use.

I did what you suggested and hallilouya, it seems to work.
I had a lot of duplicates.

thanks for your guidance.

Well I spoke too soon. It is still balled up. The parts I didn't understand in your outline was, what do you mean 'custom' libraries ? Is wire.h a custom library ? is rtc3231.h ?

When I tried to point the IDE at my sketchbook folder, I got an error saying I can't put things there because mt IDE was in the same folder. Which is my second question. You didn't mention where the arduino 1.8.2 folder should go.

I have spent hours trying to get a simple sketch to run, but I keep getting the same error : ds3231.h no such file exists.

This is how I got this so balled up before, adding library files all over the place so the IDE can find them.

I have a folder on 'D' drive called 'Arduino Sketchbook Folder'

Well I spoke too soon. It is still balled up. The parts I didn't understand in your outline were, what do you mean 'custom' libraries ? Is wire.h a custom library ? is rtc3231.h ?

When I tried to point the IDE at my sketchbook folder, I got an error saying I can't put things there because my IDE was in the same folder. Which is my second question. You didn't mention where the arduino 1.8.2 folder should go.

I have spent hours trying to get a simple sketch to run, but I keep getting the same error : ds3231.h no such file exists.

This is how I got this so balled up before, adding library files all over the place so the IDE can find them.

I have a folder on 'D' drive called 'Arduino Sketchbook Folder' :
in that folder I have the 'arduino-1.8.2-windows' folder;
and I also have 'Sketches' Folder.

In the IDE I pointed it at Sketches - no good. can't find the library.
Then I tried to point it at arduino-1.8.2. no good. error message 'the specified sketchbook folder contains your copy of the IDE, please choose a different folder for your sketchbook'.

Where should I put the ds3231.h library folder so that it can find the .h file ?

I'm just lost.

1 Like

pratto:
what do you mean 'custom' libraries ?

By custom libraries I mean any library that you wrote or modified. I also specified "3rd part" libraries, which was a typo, I actually meant to say "3rd party" libraries. That means any library you have installed, rather than the ones that are included with the Arduino IDE or hardware packages.

pratto:
Is wire.h a custom library ?

If you modified it then it is but if you're talking about the Wire library included with Arduino AVR Boards or any other hardware package then no. It's generally a bad idea to put that sort of library at {sketchbook folder}\libraries because each architecture has a different version of the Wire library. The correct version is automatically selected according to the current selection in the Tools > Board menu. Libraries located in {sketchbook folder}\libraries are given a higher include priority to allow you to override the default library with a modified version. That can be useful but can also cause problems. If you put that library in {sketchbook folder}\libraries it may cause the Arduino IDE to include the wrong version of the library, though recent IDE versions do attempt to chose the correct one by matching the library metadata (library.properties architectures property) to the architecture of the selected board. It could also cause an outdated version of the library to be used if you have updated the IDE or hardware package since you placed the library in {sketchbook folder}\libraries.

pratto:
is rtc3231.h ?

I'd need to know where you got that file from, but if I had to guess I'd say yes.

pratto:
When I tried to point the IDE at my sketchbook folder, I got an error saying I can't put things there because my IDE was in the same folder.

Do not use the IDE installation folder or a subfolder of the IDE installation folder for your sketchbook folder. You can put it anywhere else you like, but not there. That is actually a very helpful warning. You should never put any of your files in the Arduino IDE installation folder because they could be lost when you update to a new version of the Arduino IDE.

pratto:
You didn't mention where the arduino 1.8.2 folder should go.

If that's the Arduino IDE installation folder then it depends on which installation method you used. If you used the "Windows Installer" download then you should leave it at the location where you installed it to. If you want to move it you will probably need to uninstall and then reinstall. Usually the folder would be named Arduino if you installed it that way and it will be located in the Program Files folder on 32 bit Windows or the Program Files (x86) folder on 64 bit Windows. If you used the "Windows ZIP file for non admin install" then you can put the Arduino IDE installation folder wherever you like. Usually the folder will be named something like arduino-1.8.2.

pratto:
I have spent hours trying to get a simple sketch to run, but I keep getting the same error : ds3231.h no such file exists.

Where is that file?

pratto:
Where should I put the ds3231.h library folder so that it can find the .h file ?

It should be something like {sketchbook folder}\libraries\ds3231\ds3231.h. The sketchbook folder location is set in the Arduino IDE at File > Preferences > Sketchbook location. The folder of the library doesn't need to be named ds3231 but it's generally a good idea to have the folder name match the filename of the main header file of the library. The reason is if there are multiple libraries with the same file name the Arduino IDE gives preference to the one with a matching library folder name. Note that ds3231.h should not be located directly under the libraries folder, it should be in its own library folder. The Arduino 1.5 libraries format places the source files in a subfolder of the library folder named src, with a library.properties file. So in that format the library would be located somewhere like: {sketchbook folder}\libraries\ds3231\src\ds3231.h. Either format is valid with arduino IDE 1.5.0 and newer. The 1.5 format is not recognized by Arduino 1.0.x.

Well I decided to get rid of everything except my sketches and start over. I cleaned out almost all references to arduino, including deleting the installation itself. Then re-installed 1.8.2
I let it install under program files (which is where it wanted to), and it seemed to go well.

Then I set up a separate Sketchbook folder and brought in my sketches. I re-booted and ran it. It seems to be working, but I am taking it very slowly. I have stuck to the 'blink' and 'blink without delay' examples. Then modified the 'blink without delay' and saved it. that worked.

Now my plan is to slowly work my way thru known good sketches (ones that used to work for me) and see how the finding of the libraries goes. This whole mess started when I tried to load and run sketches related to the Real Time Clock. I dread trying it again. But I want to build a little yard security light, and I need RTC.

I carefully read, twice, your last response, and it made your first guide more understandable. and thank you for taking the time to help me again. I also read another thread you participated in about un-installing arduino.

There is nothing in the library folder about ds3231 or RTC. In reading up, it seems that libraries are supposed to be installed, not just copied and plopped where I think they should go. When I get to that point (and before I try it) I will probably ask for more help.

pratto:
it seems that libraries are supposed to be installed, not just copied and plopped where I think they should go.

The Arduino IDE provides two way to install libraries via the GUI:

Sketch > Include Library > Manage Libraries...
This is the library manager, an index of user submitted libraries that can be installed with a few clicks. If you have File > Preferences > Check for updates on startup checked then the IDE will show a notification when a new version is available for any library in the index that you have installed.

Sketch > Include Library > Add .ZIP library...
This can be used to install libraries that you have downloaded that are in a .zip file. It will only work if the library is in the root of the .zip file, rather than in a subfolder.

There's no magic to either of those installation methods. All they do is copy the library to {sketchbook folder}\libraries. There is absolutely no difference between the Arduino IDE doing this and you doing it manually. If I have a library in a .zip file I will use the second method because it's a bit faster. Some people will go to the trouble of zipping up libraries that are not just to install them but it's faster and easier for me to just manually copy it to the right folder in that case. If I'm trying to help someone who says they have installed a library via Library Manager I'll install it that way but usually for my own needs I don't because it supplies the latest release version and I will usually want to check the repository to see what changes have been made since the last release before deciding whether I want to go for the "beta" or opt for the possible greater stability of a release at the expense of missing out on recent improvements.

pratto:
When I get to that point (and before I try it) I will probably ask for more help.

Happy to help.

pert -

I hesitate to say it, but by following your suggestions, I re-installed the IDE, got rid of all custom libraries, put things where they are supposed to be (making sure I don't goof it up as I go), and installed .zip file libraries as I needed them - and it seems to be working fine. That was along sentence.

I even got brave and did the RTC, and it is working too. What a relief.

Now I'm on to try to figure out how to set a timer for a light each night.

Thanks again for your excellent help.

Glad to hear you're back up and running, hopefully in a more organized state too!

My situation is that I have made several hardware versions of a device that uses an Arduino Uno or Microduino Core+ for the processor, three different types of GPS receivers (so far), and two different types of barometric pressure sensors. Data collected from the GPS receiver and barometric pressure sensor are recorded to a microSD card in each of the configurations, and different shields have different SD card chip select lines. Different versions also have different serial port configurations.

I'm trying to publish this project on GitHub with at least three hardware configurations, I have different sketches and libraries for each configuration. I want my GitHub repo to be easy for others to use.

When I first started developing the three hardware variants of this project, I put all of the necessary libraries in C:\Users\username\My Documents\Arduino\libraries, where my sketchbook folder was C:\Users\username\My Documents\Arduino. That was OK for the barometric pressure sensor, because I could use one library for the BMP180 and another library for the BMP280, with no customization required. But the NeoGPS library is highly configurable via multiple header files for different protocol types (e.g., NMEA vs proprietary), variations on what NMEA sentences different receivers put out, and what information you want out of those sentences (e.g., if you don't need speed and heading, you can configure the library not to parse them). The NeoGPS library has a a src folder containing NMEA sentence processing .h and .cpp files, and subfolder under src called ublox, which contains .h and .cpp files for handling the u-blox proprietary GPS receiver protocol. In addition, using an Arduino Uno, which only has one hardware serial port, you have one GPS serial port configuration, but with the Microduino Core+, you have a second hardware serial port that you can use, which requires a different GPSPort.h configuration in NeoGPS. So I started out creating different versions of the various NeoGPS header files with different names for the different configurations. This (IMO) was a really bad idea, because if I update the NeoGPS library, the Arduino library manager replaces the existing library folder with the updated library folder, thus wiping out any custom header files I created. (Hooray for daily backups!) Moreover, it starts to get ugly (inelegant) fast.

After some research, I learned that the Arduino IDE looks for libraries in several places (the following is Windows specific): (1) the libraries folder in your sketchbook, e.g. C:\Users\username\My Documents\Arduino\libraries; (2) in the libraries folder of the Arduino IDE installation folder, e.g. C:\Program Files (x86)\Arduino\libraries; (3) in the folders below C:\Program Files (x86)\Arduino\hardware, e.g., C:\Program Files (x86)\Arduino\hardware\arduino\avr\libraries; and (4) in your sketch folder. If you want to include a library in one of the first three places, you enclose the appropriate header file name in angle brackets (<>), e.g., #include <SomeLib.h>, and if you want to include a library in your sketch (not sketchbook) folder, you enclose the header file name in double quotes, e.g. #include "MyLib.h".

In order to be able to have several different configurations of the NeoGPS library, as well as other libraries including Adafruit_BMP280_Library or Adafruit_BMP085_Library, NeoSWSerial (if using an Arduino Uno, not needed for a Microduino Core+), PString, and SDFat, the solution I eventually settled on is to have a different sketchbook folder for each configuration, containing a libraries folder and a sketch folder. I have a copy of the each of the required libraries that are normally stored in the Arduino sketchbook folder in the libraries folder of my hardware-configuration-specific sketchbook folders. The main disadvantage of this approach (in addition to multiple copies of libraries that you may need to update) is that you have to remember to point the Arduino IDE to the correct sketchbook. You can easily change the sketchbook folder that the Arduino IDE uses under File... Preferences... Sketchbook location in the Arduino IDE (I'm using version 1.8.5 right now).

Another thing I learned is that if a library has at least one subfolder containing additional source code (.h and/or .cpp) files, all of the source code files must be within a src folder inside the library folder, or the IDE won't find the source code files within the subfolder. An exception to this src folder requirement may be if the subfolder is named "utility". See the Arduino IDE 1.5: Library specfication, under the Layout of folders and files heading. I quote:

The source code found in src folder and all its subfolders is compiled and linked in the user’s sketch. Only the src folder is added to the include search path (both when compiling the sketch and the library). When the user imports a library into their sketch (from the "Tools > Import Library" menu), an #include statement will be added for all header (.h) files in the src/ directory (but not its subfolders). As a result, these header files form something of a de facto interface to your library; in general, the only header files in the root src/ folder should be those that you want to expose to the user's sketch and plan to maintain compatibility with in future versions of the library. Place internal header files in a subfolder of the src/ folder.

For backward compatibility with Arduino 1.0.x, the library author may opt to place source code into the root folder, instead of the folder called src. In this case the 1.0 library format is applied and the source code is searched from the library root folder and the utility folder...

Hope this helps someone avoid wasting the amount of time I did in making my project work and look at least a bit elegant. Wish the Arduino website discussion of libraries was a bit more clear and thorough. I don't believe one should have to peruse multiple sources, including the Arduino Tutorial and Reference, the Arduino Wiki on GitHub, and this forum to figure this stuff out.

It sounds like a difficult situation. I'm glad you found a solution and shared it.

CF20852:
I'm trying to publish this project on GitHub with at least three hardware configurations, I have different sketches and libraries for each configuration. I want my GitHub repo to be easy for others to use.

Have you published it yet? I'd like to take a look.

CF20852:
if you want to include a library in your sketch (not sketchbook) folder, you enclose the header file name in double quotes, e.g. #include "MyLib.h".

Note that if MyLib.h is not present in the local folder the other locations searched by the #include <MyLib.h> syntax will also be searched.

CF20852:
the solution I eventually settled on is to have a different sketchbook folder for each configuration, containing a libraries folder and a sketch folder. I have a copy of the each of the required libraries that are normally stored in the Arduino sketchbook folder in the libraries folder of my hardware-configuration-specific sketchbook folders. The main disadvantage of this approach (in addition to multiple copies of libraries that you may need to update) is that you have to remember to point the Arduino IDE to the correct sketchbook. You can easily change the sketchbook folder that the Arduino IDE uses under File... Preferences... Sketchbook location in the Arduino IDE (I'm using version 1.8.5 right now).

Definitely not ideal, but that may have been the best solution.

CF20852:
Wish the Arduino website discussion of libraries was a bit more clear and thorough. I don't believe one should have to peruse multiple sources, including the Arduino Tutorial and Reference, the Arduino Wiki on GitHub, and this forum to figure this stuff out.

Any specific suggestions for improvement are welcome. I think part of the problem is that documentation is typically written by people who are so familiar with the system that everything seems obvious to them and so they don't explain things as thoroughly or clearly as they might. For this reason, the perspective of someone newly encountering this information is very valuable. The Arduino team are quite receptive to suggestions for improving the documentation. The GitHub wiki is even left open to editing by anyone with a GitHub account.

I've taken a look at your project:

Rather than dealing with three sketchbooks, the way I would probably have handled the different versions of NeoGPS is to bundle the modified NeoGPS library with each sketch. So the sketch would look like this:

ArduinoUno_ATGM332D_BMP280_HikeLogger
|_ArduinoUno_ATGM332D_BMP280_HikeLogger.ino
|_src
|_NeoGPS

And your sketch includes the library like this:

#include "src/NeoGPS/src/NMEAGPS.h"
#include "src/NeoGPS/src/ublox/ubxGPS.h"

#include "src/NeoGPS/src/GPSport.h"
#include "src/NeoGPS/src/Streamers.h"

Then the sketches can all share the other libraries, which don't require any modifications.

There is another option. That is to make each version of NeoGPS a separate library with a separate name. If you change the folder name then there is no problem with Library Manager updates overwriting the modified library. If you change the header file name then you will be able to control which library is included by your code. The advantage of this is that it would permit you to merge all three of your sketches into one. The you can have a configuration macro at the top of the sketch:

// Uncomment one of the following lines to configure the sketch:
// #define ARDUINOUNO_ATGM332D_BMP280
// #define ARDUINOUNO_DUINOPEAKNEO-M8N_BMP280
// #define MICRODUINO_NEO-6M_BMP180

Then anywhere there is code that differs between the versions:

#if defined(ARDUINOUNO_ATGM332D_BMP280)
// ARDUINOUNO_ATGM332D_BMP280 specific code here
#elif defined(ARDUINOUNO_DUINOPEAKNEO-M8N_BMP280)
// ARDUINOUNO_DUINOPEAKNEO-M8N_BMP280 specific code here
#else
// MICRODUINO_NEO-6M_BMP180 specific code here
#endif

This is a good option if you have a lot of code that's shared between the three sketches, otherwise when you want to make a change it get's really tiresome having to do it 3X every time. The disadvantage is all that preprocessor code gets pretty messy. I haven't looked over the sketch code closely enough to say whether this is worth it.

Yes, if in doubt just make all problematic dependencies 'local' to the sketch file. It just means that all version control is going to be manual. I do this most of the time otherwise the arduino library management 'does my head in'.