Custom libraries

Projects Programming
1

I have a function that is used by a number of my sketches. I would like to put the structure definitions and the function into #include files and then replace the copies that are in my sketches with #include statements. For reference, I am using Arduino IDE 1.8.13 on Windows 10.

To that end, I created a created a directory called ..\mySketches\library\comhdr and put a file in this directory that is called comhdr.h. It contains the structure definitions. In a sketch I replaced the structure definition with the statement #include “comhdr.h”. I then compiled the sketch and it completed without error.

I attemtpted the same procedure with the function: I created a directory ..\mySketches\library\comfunc and put a file in this directory that is called comfunc.c. It contains the function. In the sketch I replaced the function with the statement #include “comfunc.c”. I then compiled the sketch. The compile fails, saying that the file comfunc.c cannot be found.

I then changed the include to #include “comfunc.h” and COPIED the comfunc.c file into comfunc.h. The subsequent compile failed with the following error messages: exit status 1, Error compiling for board Arduino Uno.

I then erased the comfunc.c file, leaving the comfunc.h file. This time the compile failed stating that the comfunc function was undefined. I then added the function definition to the comhdr.h file and the sketch compiled without error.

I have the following questions:

Q1: The procedure that worked included the code for the function in a “.h” file. In my experience with other environments, code that is included with a #include statement generally is placed in a file called xxx.c, or xxx.cpp, not xxx.h. Is this the correct way to include code in an external file in the Arduino IDE, or is there a better way?

Q2: Along the same line as Q1: why did the compiler fail to find the file named comfunc.c, but it had no problem finding comfunc.h?

Q3: Is there a way to put both the definitions file and the code file into a single library subdirectory?

Q4: Why did the compiler fail with exit code 1 when the library directory contained both comfunc.c and comfunc.h files, but does not fail after I erased the “.c” file?

Q5: The Arduino compiler does not require me to predefine the functions in an Arduino program. I have found this to be quite convenient, as other compilers require pre-definition. However, when I include a function using #include, I am forced to predefine the function. Is there an explanation for this?

2

A1. See My Reply #5 in this Thread for a basic guide to what should go into .h files and what should go into .cpp (or .c) files. Also, I'd recommend against using .c files unless there's a sound reason why you want the code to be compiled as 'C' vs 'C++'.

A2. I would recommend putting both the .h and .cpp files in single folder with the custom library's name. Put that folder in the "libraries" folder inside your sketchbook folder. You should also study the Arduino Library Specification.

See A2.

A4. It won't if you do it correctly.

Arduino's automatic function prototype generator is a crutch that only works for .ino files. And it fails sometimes. IMO, you are always better off providing your own function prototypes.

2 Likes
3

Thank you for your reply and for your references. I read over the material you referenced and attempted to follow the directions, but I'm still having problems and solicit further advice.

In my ..\libraries directory I created the following...

\cdjlib
In this subdirectory I put a file called library.properties. I copied this file from another library and then edited it create my custom profile.

\cdjlib\src
In this subdirectory I put the comhdr.h file, and the comfunc.cpp file.

In the the program .ino file I replaced the header lines with #include "comhdr.h" and I replaced the function lines with "#include "comfunc.cpp".

The compile fails with "Error 1" error I mentioned previously. However, if I rename the file containing the function to "comfunc.h" (and change the include to match), then the compile works ok.

With this configuration I now have both files in the \src subdirectory, and under my custom library subdirectory. So far so good. However, for a good compile the function must be in a file named xxx.h, not xxx.cpp file. What am I doing wrong?

| gfvalvo
October 26 |

  • | - |

cdj15:

Q1: The procedure that worked included the code for the function in a “.h” file. In my experience with other environments, code that is included with a #include statement generally is placed in a file called xxx.c, or xxx.cpp, not xxx.h. Is this the correct way to include code in an external file in the Arduino IDE, or is there a better way?

A1. See My Reply #5 in this Thread for a basic guide to what should go into .h files and what should go into .cpp (or .c) files. Also, I'd recommend against using .c files unless there's a sound reason why you want the code to be compiled as 'C' vs 'C++'.

cdj15:

Q2: Along the same line as Q1: why did the compiler fail to find the file named comfunc.c, but it had no problem finding comfunc.h?

A2. I would recommend putting both the .h and .cpp files in single folder with the custom library's name. Put that folder in the "libraries" folder inside your sketchbook folder. You should also study the Arduino Library Specification.

cdj15:

Q3: Is there a way to put both the definitions file and the code file into a single library subdirectory?

See A2.

cdj15:

Q4: Why did the compiler fail with exit code 1 when the library directory contained both comfunc.c and comfunc.h files, but does not fail after I erased the “.c” file?

A4. It won't if you do it correctly.

cdj15:

Q5: The Arduino compiler does not require me to predefine the functions in an Arduino program. I have found this to be quite convenient, as other compilers require pre-definition. However, when I include a function using #include, I am forced to predefine the function. Is there an explanation for this?

Arduino's automatic function prototype generator is a crutch that only works for .ino files. And it fails sometimes. IMO, you are always better off providing your own function prototypes.

4

Give the two files the same name, differing only in the ".h" or ".cpp" extension. DON'T #include the .cpp file in your .ino file, just the .h file!!!

I that doesn't work, post all three files here (.h, .cpp, .ino).

5

Thank you very much for your help with this issue. I followed your advice, but the problem continues. A short background of this sketch might be helpful.

I have several Arduino systems that include a data collection board with a Real Time Clock (RTC). I have found that the RTC chips typically run fast, anywhere from 5 to 8 or so seconds per day. I developed a sketch called fixclock.ino that will do several things. The actions are compiled choices. It can set the RTC, or read the RTC and report the status, including the seconds per day error of the RTC, or run an array of tests to verify the logic of the RTC correction logic.

The RTC correction logic is contained in a function called getTime. This function reads the RTC and stores the corrected RTC values in a global structure. The design is that the getTime function is included in the fixclock sketch and any sketch that I run on any board with an RTC. Important values that are needed by getTime to correct the RTC (the date and time that the RTC was last set, and the daily correction value) are stored in EEPROM by the fixclock sketch and read by the getTime function. The getTime function contains a significant amount of logic that is common to fixClock and any other sketch that has an RTC. But due to a few variations, there are some #if defined inclusions and exclusions in the getTime function.

My use of these boards and the correction of the RTC error has been an evolutionary process over a few years. Initially I included the data structures in global space. I copied and pasted the structures and the getTime function from fixclock.ino into any new sketch that I developed. This is a well recognized case of bad practice. As the number of projects grew, and the number of boards with an RTC grew, the bad practice finally got to me and I decided to create #include files. That brings the story to the current time.

Since this is a compile issue and not an execution issue, to simplify the analysis of the problem I renamed the fixClock sketch to test2.ino and stripped most of the irrelevant logic out of the fixclock sketch. All of the original functions and function calls within it remain. This reduced sketch demonstrates the problem. A #include “cdjCurTod.h” statement is included in the front matter of the sketch, and the cdjCurTod.h file is in the ..\libraries\cdjRtcLib\src\ directory.

- When the cdjCurTod.cpp file is in the \src directory, and there is no #include for this file in the .ino file, then the compile fails with “exit status 1”. The compile also fails with “exit status 1” if an #include is included for the .ino file.

- When the cdjCurTod.cpp file is renamed to cdjGetTime.h, and an #include “cdjGetTime.h” is included in the .ino file, then the compile completes without error. But this isn’t how things are supposed to work.

And there is more. One of many sketches that use an RTC, and would benefit from the #include configuration, is called sslog.ino. This sketch receives a line of data from another Arudino over the software serial interface, applies a timestamp, and writes the line to an SD flash disk file. I replaced the structures and the getTime function with #include statements. However, using the new #include configuration, I have not been able to find any combination that results in a successful compile. As with the test2.ino sketch, a #include “cdjCurTod.h” statement is included in the front matter of the sslog.ino sketch, and the cdjCurTod.h file is in the ..\libraries\cdjRtcLib\src directory.

- When the cdjCurTod.cpp file is in the \src directory, and there is no #include for this file in the .ino file, then the compile fails with “exit status 1”. The compile also fails with “exit status 1” if an #include is included for the .ino file. This is consistent with the behavior of the test2 sketch.

- However, unlike to test2 case, when the cdjCurTod.cpp file is renamed to cdjGetTime.h, and an #include “cdjGetTime.h” is included in the .ino file, the compile still fails with “exit status 1”.

As with the fixclock sketch, to simplify the analysis of this issue I copied the sslog.ino code into test3.ino and removed most of the irrelevant code. The test3.ino file is also included in the attached zip file.

I created a zip file for the files instead of including them in the forum post because they exceed the character limit that the Arduino forum site places on postings.

customlibs.zip (17.7 KB)

6

First error

c:\Users\bugge\OneDrive\Documents\Arduino\libraries\cdjCurTod\cdjCurTod.cpp: In function 'void getTime()':
c:\Users\bugge\OneDrive\Documents\Arduino\libraries\cdjCurTod\cdjCurTod.cpp:6:3: error: 'adjTod_t' was not declared in this scope
   adjTod_t  adjTod;
   ^~~~~~~~

You need to include your .h file in you .cpp file as well.

After that

c:\Users\bugge\OneDrive\Documents\Arduino\libraries\cdjCurTod/cdjCurTod.h:21:3: error: 'byte' does not name a type
   byte               cpuBoardId;                       // my board id for this arduino board
   ^~~~

The easiest ways is to Arduino.h in your .h file; that will make anything that is usually known in a sketch also known in a library.

Further it's advisable to use include guards or a #pragma once in you .h file. You can look those up. It prevents problems about multiple definitions of variables when you have a project where you include the the .h file in multiple files that are part of the sketch.

This does not solve all your problems, there is still (and maybe more)

c:\Users\bugge\OneDrive\Documents\Arduino\libraries\cdjCurTod\cdjCurTod.cpp:19:9: error: 'epStore' was not declared in this scope
     if (epStore.rtcChipId == DS1307) now = rtc1307.now();
         ^~~~~~~

because the declaration was commented out in the .h file.

If you're still stuck after that you can post your new library file here using code tags instead of attaching; they are still small enough and it's easier for users. Also post the error message using code tags; just in case, see https://forum.arduino.cc/t/how-to-get-the-best-out-of-this-forum/679966#posting-code.

7

tl;dr

Then don't post the complete code, especially if it's long, messy, and convoluted. Instead post a Minimal Reproducible Example (MRE). This is the smallest possible complete code that demonstrates the problem at hand, and only that problem.

8

- I added the .h file to the .cpp file.

- I also added guards to the #include files. It was my intent to do this, but I was admittedly sloppy and hadn’t done it yet.

- The compiler complained about a variable of type “byte”. It appears that the data type “byte” is acceptable in an .ino file, but not in a #include file. I don’t understand why. For this exercise I changed the “byte” variables to “unsigned char” and avoided the error. I think I could also #define “byte” to be “unsigned char” to avoid the labor of editing.

- The epStore variable declaration is commented out in the .h file because it is declared as a global variable in the .ino file, immediately after the #include cdjCurTod.h. I will delete the commented declaration in the .h file to avoid confusion.

- I added #include <Arduino.h> to the .ino file, but this didn’t make any difference, but I left it in anyway. I have never heard of this .h file. The literature says that this is usually automatically included by the IDE, so that’s probably why I have not encountered it before. My explicit inclusion is probably redundant, but guard tags take care of that.

The current problem is that the compiler doesn’t seem to recognize variables that are defined as global in the .ino file.

I think I see why, but I’m not sure how to fix it. From the compiler output, it appears that the compiler encounters the #include cdjCurTod.h in the .ino file. This .h file includes the data type declarations that are needed to define variables later in the .ino file. So far so good.

The compiler proceeds to processes the .h file. As part of this processing, and without explicit instructions to do so, the compiler also processes the cdjCurTod.cpp file. The .cpp file includes the getTime function which references global variables that are defined in the .ino file. However, these definitions follow the include of the .h file in the .ino file and have not yet been processed by the compiler, thus resulting in a “not declared in this scope” error.

9

Is there any reason you haven't complied with this simple request?

10

I guess that's too much to ask. Take a look at this structure:
Main.ino:

#include "Main.h"
#include "Library.h"

uint32_t globalVar1;
float globalVar2;

void setup() {
  libraryGlobalVar1 = 0;
  libraryGlobalVar2 = 2.7183;
  libraryFunction();
}

void loop() {
}

Main.h:

#ifndef MAIN_H
#define MAIN_H

#include <Arduino.h>
extern uint32_t globalVar1;
extern float globalVar2;

#endif

Library.h:

#ifndef LIBRARY_H
#define LIBRARY_H

#include <Arduino.h>
extern uint32_t libraryGlobalVar1;
extern float libraryGlobalVar2;

void libraryFunction();

#endif

Library.cpp:

#include "Main.h"
#include "Library.h"

uint32_t libraryGlobalVar1;
float libraryGlobalVar2;

void libraryFunction() {
  globalVar1 = 100;
  globalVar2 = 3.1416;

  // Do stuff here

}
11

Hi @cdj15.

It is explained here:

https://arduino.github.io/arduino-cli/latest/sketch-build-process/#pre-processing

  • If not already present, #include <Arduino.h> is added to the sketch. This header file (found in the core folder for the currently selected board) includes all the definitions needed for the standard Arduino core.

The definition of the custom byte type is in Arduino.h. For example, here you can see it in the core of the "Arduino AVR Boards" platform:

As you noted, an #include directive for that file is automatically added to the code of the .ino sketch files, you can use the type in the .ino files of the sketch without needing to do anything special. However, that is not so for files of any other type:

No pre-processing is done to files in a sketch with any extension other than .ino or .pde.

So if you want to use the objects declared in Arduino.h in a .h or .cpp file, you must add an #include directive for that header to the file.

It is not necessary, but it also probably won't do any harm. Those who are already fluent in C++ might even prefer to do this if they find the automagical sketch preprocessing performed by the Arduino build system, which allows writing sketch code that is not valid C++, to be confusing.

Yes, all .cpp files present in the root of the sketch folder, and under the src subfolder are compiled.

12

See quote below. Explanation given by @ptillisch in post #11.

13

Thank you for your reply. I’m getting closer to the goal of using #includes for this project, but there are still a couple of issues.

- It appears that global variables in the .ino file that are referenced in a library #include .cpp file must be declared as “extern” in the library .h file. Is this correct? If so, this seems like the compiler doesn’t recognize globals as truly global within a single compile.

- I wish to use #define to define a variable in the .ino file and use #ifdef to conditionally compile statements in the library .h file and in the library .cpp file. The #ifdef works fine in the .h file. However, the #defined variable is always not defined in the .cpp file. The following intentional compile error demonstrates the issue. I defined FIXCLOCK in the .ino file. If FIXCLOCK was recognized as defined when the .cpp file is compiled, then the statement fff=ggg would be flagged with an error. Is there a way to get #defines in the .ino file to be recognized in the library .cpp file?

myTestLib.h

#ifndef myTest_hdr
#define myTest_hdr
int testInt = 0;
void testFunc();
#endif // myTest_hdr

myTestLib.cpp

#ifndef myTest_code
#define myTest_code
#include "myTestLib.h"

void testFunc() {
#ifdef FIXCLOCK
  fff=ggg;
#endif

#ifndef FIXCLOCK
  hhh=iii;
#endif
  } // end testFunc function
#endif // myTest_code

test1.ino

/* test #includes with #defines */
#define FIXCLOCK
#include "myTestLib.h"
void setup() {
Serial.begin(9600);
Serial.print("Begin Test1...\n");
Serial.print("End of setup\n");
} // end setup

void loop() {
} // end loop

The results of the compile…

Compiling libraries...

Compiling library "myTestLib"

"E:\arduino\arduino-1.8.13\arduino-1.8.13\hardware\tools\avr/bin/avr-g++" -c -g -Os -std=gnu++11 -fpermissive -fno-exceptions -ffunction-sections -fdata-sections -fno-threadsafe-statics -Wno-error=narrowing -MMD -flto -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=10813 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR "-IE:\arduino\arduino-1.8.13\arduino-1.8.13\hardware\arduino\avr\cores\arduino" "-IE:\arduino\arduino-1.8.13\arduino-1.8.13\hardware\arduino\avr\variants\standard" "-ID:\cdj\arduino\cdjprojects\libraries\myTestLib\src" "D:\cdj\arduino\cdjprojects\libraries\myTestLib\src\myTestLib.cpp" -o "C:\Users\cdj\AppData\Local\Temp\arduino_build_955767\libraries\myTestLib\myTestLib.cpp.o"

D:\cdj\arduino\cdjprojects\libraries\myTestLib\src\myTestLib.cpp: In function 'void testFunc()':

D:\cdj\arduino\cdjprojects\libraries\myTestLib\src\myTestLib.cpp:11:3: error: 'hhh' was not declared in this scope

hhh=iii;

^~~

D:\cdj\arduino\cdjprojects\libraries\myTestLib\src\myTestLib.cpp:11:7: error: 'iii' was not declared in this scope

hhh=iii;

   ^~~

Using library myTestLib at version 1.0.0 in folder: D:\cdj\arduino\cdjprojects\libraries\myTestLib

exit status 1

Error compiling for board Arduino Uno.
14

The trouble is you don’t yet understand how multi-file projects are compiled and linked. Each .cpp file and .ino file (which is really just a precursor to a .cpp file) is compiled as a separate compilation unit. They know nothing about each other except for the hints provided by constructs like extern and function prototypes in the #include(d) .h files. These hints provide just enough information to tell the compiler that the referenced objects exists and what its interface look like. They don’t tell where it is or what it does. However, these objects must be defined in one and only one compilation unit. It’s the linkers job to take all this information and stitch together a cohesive project.

That's and easy one. You can't (mainly for the reason above). Don't even try. Many a newbie here in the forum has needed to be disabused of the idea.

BTW, #define does not define a variable. It's just a simple text substitution macro.

15

A user friendly library API does not typically rely on globals. Take some time to study the API of existing libraries to understand how to create an intuitive interface.

You might find this tutorial useful:

https://docs.arduino.cc/learn/contributions/arduino-creating-library-guide/

Although there can be specific use cases where making such configuration via the sketch might be desirable, it is fairly rare. In most cases, any such configuration is only needed for library development (e.g., enabling debug output), or for very advanced niche use cases. In those cases, it can be done by editing the library code instead of via the sketch code.

If you find yourself wanting such a thing for a common usage of the library, investigate whether there is an alternative approach that does not use the preprocessor. Any preprocessor-based interface to a library is very user hostile and can also make the library codebase very unfriendly to the developer. The preprocessor should only be used in cases where there is absolutely no better alternative.

16

As always, I appreciate your help.

Yes, I understand that #define is defining a macro, not a variable. I considered using the term “macro” in my post, but chose the term “variable” instead, perhaps inadvisably, fully recognizing the difference. Technically speaking, I think that a statement such as “#define x” defines a null macro, and “#ifdef” provides the ability to detect the prior definition, or not, of a macro, whether null or not.

I understand that the pieces of a multi-file project are compiled separately, and that is why extern’s are required. However, with this in mind, the part that has caused confusion for me is this: the .cpp file is implicitly selected for compilation as a result of the content of the #include of the .h file. Without a #include for the .h file in the .ino file, the .h and .cpp files would not be included in the compile. However, with the #include of the .h file, these files are included. So far so good.

The confusing part is that for this separate compile of the #include files, #define’s that are set in the .ino file ARE recognized when preprocessing the .h file, but these same #define’s ARE NOT recognized by the same single preprocessing step when preprocessing the .cpp file.

Ok, like it or not, the way this system works is that they are recognized in the .h file and not the .cpp file.

I got it.

As I noted earlier in my description of why I think a library would be useful for these projects, there is a large amount of code in a function that is common to (1) the “master” project that is used to read and set the RTC, and (2) many “slave” projects that don’t need and shouldn’t have, very small selected parts of the common function, because these parts are unique to the master. Separately extracting and isolating these pieces from the common function is not practical. That is why it is necessary for some parts of the common function to be excluded in the slave projects. Furthermore, the exclusion is essential in sketches that are pushing the memory limits of Arduino R3.

So, even thought it might seem strange to me that #define’s in the .ino are recognized in the .h and not in the .cpp, that’s the way it works. Thus, using #ifdef in the .cpp file appears to be out. I got it.

Is there any other way to achieve the conditional compile within the .cpp file that is controlled by something in the .ino file?

17

You can think of an #include directive as being replaced by the contents of the file specified by the #include directive. Do you now see why it is that macros defined in the code of the .ino file may affect the code in the .h file.

There are some important things to keep in mind:

This will only be true for macros defined above the #include directive in the .ino code.

This will only be true when compiling the code of the header file as it is part of the "translation unit" of the .ino file. The code from the header file may also be compiled as part of other translation units that contain an #include directive for the header, and the macros defined in the .ino translation unit will have no effect on the header file code when it is compiled in those other translation units, as you noticed:

Note that this is not anything specific to Arduino sketch build system. This is simply how C++ works.

You may not need conditional compilation. The compiler is pretty good about optimizing out things that are not used.

If you absolutely do need the ability to control the code of a library via macros defined in the sketch code, you will need to make a header-only library, where all relevant source code is in the header file instead of .cpp files. But please take heed of the warning I gave in my previous reply regarding unnecessary use of the preprocessor in your code, and especially as API.

You can see an example of the header-only approach here:

18

@cdj15

Minor remark, for robustness and maintenance sake, use the most explicit integer datatypes in your code. E.g.

  • uint8_t or int8_t instead of byte
  • int16_t or int32_t instead of int

Especially the latter is important, I have seen it failing in code (including my own) as some boards implement int as 16 bit (AVR) where others implement it as 32 bit (ESP32).

my 2 cents.

19

But, even with that, @cdj15 needs to realize that it's impossible to affect the compilation of one translation unit by defining a macro in another translation unit. Meaning you can't affect the compilation of a .cpp file by defining a macro in the .ino file. And, vice versa. There is no way to do this. Period.

If you must affect the compilation of multiple translation units using a single macro, then that macro must be #define(d) in a single, common .h file that's #include(d) in all translation units.

Also, just a note on terminology ... .h files are not compiled! They are #include(d) into .cpp files that are compiled.

20

Thank you for your assistance and guidance.

As I noted in the beginning of this thread, I have a program called fixclock that I use to manage the RTC for multiple Arduino systems. This program sets the RTC, reads and reports the current RTC time drift error based on the fixclock compile time, and a few other things. A function I call getTime() in this program performs some rather complicated RTC correction logic that is needed by all of my other Arduino boards that have an RTC. However, there are minor code differences in the gettime() function that is in the fixclock program, which is “full function”, and the gettime() function in all other programs, which only read and use the adjusted clock time. This difference is controlled by #ifdef statements in the gettime() function.

Alternative methods of dealing with this difference require the inclusion of unneeded variables and unneeded code in all of the application sketches. Some of these sketches are very tight on variable memory or code memory, or both. This contributed to my desire to use #ifdef’s to control this difference.

For program maintenance reasons it was my desire to have a single copy of the gettime() function and use #include statements to include the code in programs that use an RTC: fixclock as well as all other sketches.

I have now created 2 .h files that achieve this objective. One includes typedefs and structure definitions, and the other includes the code for the gettime() function.

With the help you have provided, I understand much more about the pre-processing that occurs at compile time, and the structure of libraries in the Arduino IDE. I also understand that most of this processing is not unique to the Arduino environment. I realize that including code in a .h file may not be the most desirable, and not recommended in the common case, but it appears to be the only way to achieve the desired result for my perhaps uncommon case. I also now have a better understanding of the pre-processing compilation of separate .cpp files that occurs for library files. I understand why the inclusion of #ifdef’s in the gettime() function of a separate .cpp file cannot be made to work in the conventional library structure. It appears that my objective doesn’t quite fit the typical “library” case, and therefore I must use some variation in order to achieve my desired result. Using a .h file to contain the gettime() function provides the ability to use #ifdefs to customize this function.

Also:

- thank you for your comments about the “byte” type. As I apply maintenance to my various programs I will endeavor to remove my use of byte.

- I am well aware of the problem of “how big is an int?”. Most of the data that I collect on my data logger boards with an Arduino use an int variable, where int is 2 bytes. This data is typically digested by a Windows C program where int is 4 bytes.

Thanks again.