Meaning of libraries

I’m trying to compile some delivered Arduino software from somebody who has walked off the project.

I am completely confused by Arduino libraries. I’m used to makefiles where a -I directive tells the compiler where to find include files, and all c modules are explicitly compiled instead of just being found.

I installed the Arduino API for Linux at /big/arduino

Here are the include headers I need

#include <SoftwareSerial.h>
#include “task.h”
#include “portable.h”
#include “semphr.h”
#include <SPI.h>
#include <Wire.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1306.h>

esp32_littledisplay:1:28: error: SoftwareSerial.h: No such file or directory
compilation terminated.
exit status 1
SoftwareSerial.h: No such file or directory

I find the file SoftwareSerial.h here.

/big/arduino/arduino-nightly/hardware/arduino/avr/libraries/SoftwareSerial/src/SoftwareSerial.h

How do I tell the Arduino IDE to find it there? What is the meaning of the libraries directory inside the IDE directory

There are also interesting directories in my home directory

/home/francis/Arduino/libraries only contains a readme
/home/francis/.arduino15/packages/esp32/hardware/esp32/1.0.2/libraries contains loads of libraries

When I comment out #include <SoftwareSerial.h>

Multiple libraries were found for “SPI.h”
esp32_littledisplay:7:26: error: Adafruit_GFX.h: No such file or directory
compilation terminated.
Used: /home/francis/.arduino15/packages/esp32/hardware/esp32/1.0.2/libraries/SPI
Multiple libraries were found for “Wire.h”
Used: /home/francis/.arduino15/packages/esp32/hardware/esp32/1.0.2/libraries/Wire

I have not explicitely installed any libraries. I have no idea where the directory /home/francis/.arduino15/ came from

Right now I’m assuming that the libraries directory inside the IDE installation exists for installation convenience but are not actually installed.

But when I try Sketch->Include Library->Manage Libraries I see no library SoftwareSerial, but many libraries containing the word serial

Is a .ino file the top level of a sketch which would mean project or build in other development environments.
It seems just a .cpp file to me, not a project file. But libraries are installed for a sketch, not for the entire environment?

So my basic question is which libraries do I need to install and how to get the headers above. I know I get Adafruit_SSD1306 from https://github.com/adafruit/Adafruit_SSD1306
How does the Arduino IDE find the modules like SPI and Wire to compile and link in once I’ve actually made it capable of finding header files.

and your MCU is ESP32?

SoftwareSerial library is bundled with the boards package for the MCU architecture

Yes my target hardware is the ESP32. I installed it as a board package.
tools->board shows "ESP32 Dev Module"

So why doesn't the compiler find SoftwareSerial.h is it's part of the boards package.
I know the original developer used Windows instead of Linux but surely that makes no difference?

fcagney:
Yes my target hardware is the ESP32. I installed it as a board package.
tools->board shows "ESP32 Dev Module"

So why doesn't the compiler find SoftwareSerial.h is it's part of the boards package.
I know the original developer used Windows instead of Linux but surely that makes no difference?

are you sure the code is for esp32? the Arduino boards package for esp32 doesn't have SoftwaeSerial library. esp32 has more Serials so SoftwareSerial is not needed

I think I made a mistake in installing the nightly build.
I have now downloaded arduino-1.8.9-linuxaarch64.tar.xz
I'll remove all arduino directories and try again with this installation
I know after installing the arduino IDE I must add ESP32 Dev as a board.

Now since the software I was delivered relies on and works with SoftwareSerial.h which library must I install to get SoftwareSerial.h?

I somehow understand that properly installing a library makes its include files available to the IDE. But I'm still mystified as to how you include the needed source code libraries as part of the build?

fcagney:
I somehow understand that properly installing a library makes its include files available to the IDE.

Generally this is true. There are a couple libraries folders that the Arduino IDE searches for files that match your #include directives. Once found, the -I compiler flag is generated accordingly.

However, the IDE has a feature that allows libraries to be bundled with an Arduino hardware package. This is useful for architecture-specific libraries. For example, each hardware package has its own version of the SPI library. The APIs are the same between all the SPI libraries, but the low level code is very different from one hardware package to another due to the microcontrollers having different architectures. SoftwareSerial is one of these architecture-specific libraries. The Arduino IDE only searches the libraries bundled with the hardware package for the board you have selected from the Tools > Board menu.

fcagney:
I find the file SoftwareSerial.h here.

/big/arduino/arduino-nightly/hardware/arduino/avr/libraries/SoftwareSerial/src/SoftwareSerial.h

How do I tell the Arduino IDE to find it there?

Short answer: you can't.
/big/arduino/arduino-nightly/hardware/arduino/avr is the location of the Arduino AVR Boards installation. /big/arduino/arduino-nightly/hardware/arduino/avr/libraries/SoftwareSerial is the software serial library written specifically for the AVR architecture. So when you are compiling for an ESP32 board, that library won't be seen. Even if you installed that SoftwareSerial library to one fo the standard libraries folders so it would be found during the dependencies search, you would just end up with a compilation error because that library is fundamentally incompatible with the ESP32 architecture.

fcagney:
Now since the software I was delivered relies on and works with SoftwareSerial.h which library must I install to get SoftwareSerial.h?

You could search around to see if some 3rd party library author wrote an implementation of SoftwareSerial for ESP32. Much better would be to simply modify your code to use one of the extra hardware serial interfaces on your ESP32. Hardware serial is always superior to software serial.

fcagney:
esp32_littledisplay:7:26: error: Adafruit_GFX.h: No such file or directory
compilation terminated.

Some libraries come bundled with the Arduino IDE and the Arduino hardware packages. However, there are thousands more official and 3rd party libraries that you need to install to use. The Adafruit GFX library is one of these. In this case, it can easily be installed via Sketch > Include Library > Manage Libraries.

More information on installing Arduino libraries:
https://www.arduino.cc/en/guide/libraries

fcagney:
Multiple libraries were found for "Wire.h"
Used: /home/francis/.arduino15/packages/esp32/hardware/esp32/1.0.2/libraries/Wire

This is bad news. It seems you've already been mucking around with your installation. The Wire and SPI libraries are architecture specific so you should only have a single installation of these libraries that can be seen by the Arduino IDE at any one time. For some reason, you have multiple installations of these libraries visible to the Arduino IDE's library dependencies search. It's true that the IDE picked the right library, but installing copies of the SPI and Wire libraries to the standard libraries folders is a very bad idea because those libraries are architecture specific, so this could cause the version of those libraries that was written for a different architecture to be used.

fcagney:
I have not explicitely installed any libraries.

Then how do you have copies of the SPI and Wire libraries installed to the standard libraries folder?

fcagney:
I have no idea where the directory /home/francis/.arduino15/ came from

This is the location where the hardware packages you install via Boards Manager (Tools > Board > Boards Manager) are stored.

fcagney:
I'm trying to compile some delivered Arduino software from somebody who has walked off the project.

I'm guessing they wrote the code for some architecture other than ESP32. Likely it was AVR.

fcagney:
Is a .ino file the top level of a sketch which would mean project or build in other development environments.
It seems just a .cpp file to me, not a project file.

It's not a project file. There is no project file. After some minor preprocessing, the .ino files of your sketch are compiled as C++:

fcagney:
But libraries are installed for a sketch, not for the entire environment?

It is possible to bundle libraries with a sketch, and those bundled libraries will only be usable from that sketch, but this is not the case with your sketch. It's also possible to add .h, .cpp, .c, .S files to your sketch. You can use #include directives for those .h files as well as to bring in library dependencies.

fcagney:
I know the original developer used Windows instead of Linux but surely that makes no difference?

Correct. The Arduino IDE works exactly the same no matter which OS you're using.

fcagney:
I think I made a mistake in installing the nightly build.
I have now downloaded arduino-1.8.9-linuxaarch64.tar.xz

I think that is a good move. The nightly build is primarily intended for beta testing. It may not be as stable as the production releases of the Arduino IDE. However, the use of the nightly build has nothing to do with your problems with the SoftwareSerial library and switching to Arduino IDE 1.8.9 will not solve this.

fcagney:
I'm still mystified as to how you include the needed source code libraries as part of the build?

Just install the library, add the #include directive for the library header file to the sketch, and the Arduino IDE automagically handles the rest.

Right Pert, I see you have a lot of stars already. I'd love to give you another one.

That is truly an outstanding answer and if I can't figure things out then I might as well give up

Thankyou very much for making such an effort.

But one more question

/home/francis/Arduino/libraries
/home/francis/.arduino15/packages/esp32/hardware/esp32/1.0.2/libraries

Both Arduino and .arduino15 were auto created.

Does .arduino15 contain libraries installed by targetting ESP32
and Arduino libraries explicitly installed?

/home/francis/Arduino/libraries
is the folder for libraries installed with Library Manager or manually installed libraries

/home/francis/.arduino15/packages/
is a folder where boards packages are installed with Boards Manager