Go Down

Topic: Precompiling a big library? (Read 1 time) previous topic - next topic

cheche_romo

Hello I don't know if this is the right sub-forum, but I am trying to look for a solution for sharing a library with some friends. I don't know how to call it , but lets call it a multi-platform library, with this I mean that there is a lot of preprocessor code inside the library so it can compile on different platforms such as Arduino ide, PSoC creator ide or even plain c++.

One of the problems is that the library is now of a considerable size. The chart below was obtained from cloc:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
C++                             39           2193           596               9180
C/C++ Header             105           1391           439               4898
-------------------------------------------------------------------------------
SUM:                           144           3584          1035             14078
-------------------------------------------------------------------------------

This means there are currently 144 files involved, when building for Arduino I have to manually copy each file from the library directory, to a new directory so all of the files are on the same directory so Arduino ide can find them when compiling. This becomes pretty annoying when debugging code.

So I want to know if there is a better solution for this problem. Maybe precompiling the library? Is there a way to enable recursive header searching inside the Arduino libraries? Which is the right way to manage big libraries on Arduino ide?

pert

It is possible to use pre-compiled libraries in the Arduino IDE. See the documentation of the "precompiled" field of library.propertied here for details:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format

cheche_romo

It is possible to use pre-compiled libraries in the Arduino IDE. See the documentation of the "precompiled" field of library.propertied here for details:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#libraryproperties-file-format
Great so precompiled libraries works, but is there any book or reading on these topics?, I mean I have really no Idea on how to accomplish this, should I learn about makefiles?

I found this link that is the documentation on GNU, maybe I should make a different make file for each of the libraries build, maybe this way I can have only one directory structure.

But how should I go about compiling the library? , because the library includes "Arduindo.h", when I try to compile using gcc -c MyLibrary.cpp I starts throwing me a lot of errors, And I know that this library actually works and compiles on Aruino IDE.

If anyone could point me in the right direction I would really appreciate it, this library has been on stand by for months now because of this and other problems, and I would love to continue developing it.

pert

The way this works is you need to compile the library for each target and create a specific folder structure to store the compiled files. So you would need to know which targets you want your library to support.

There is a problem with support for the precompiled feature on some Arduino hardware package. Here it is described for the Arduino AVR Boards package:
https://github.com/arduino/Arduino/issues/7005
I thought that particular one had been fixed via:
https://github.com/arduino/ArduinoCore-avr/pull/52
but I just saw that it ended being reverted because it caused a problem. The solution is known and described here:
https://github.com/arduino/ArduinoCore-avr/pull/52#issuecomment-554947932
but nobody has implemented it yet.

I have never used this feature, but I believe there is some information on how to use it buried in an issue report in one of Arduino's GitHub repositories. I was looking for it when I found the issue I described above. I can continue looking and probably find it, but I thought I would check with you to see if you're still interested. It may be that requiring the users of your library to find the platform.txt file associated with the board they're compiling for and modify to make the precompiled feature work will not be feasible. That would depend on who the target users are. So let me know what you think.

cheche_romo

I have never used this feature, but I believe there is some information on how to use it buried in an issue report in one of Arduino's GitHub repositories. I was looking for it when I found the issue I described above. I can continue looking and probably find it, but I thought I would check with you to see if you're still interested. It may be that requiring the users of your library to find the platform.txt file associated with the board they're compiling for and modify to make the precompiled feature work will not be feasible. That would depend on who the target users are. So let me know what you think.
So as I understood I need to create a .a file for each of the targets I intend to use. Then create a library that includes those .a files. And finally modifying each of the platform.txt for each of the targets I intend to use.

This sounds ok until the last part, because if you update the IDE, you'll need to redo all.

But there are some steps in the way of creating those .a files that I don't fully understand. How do I compile the library from the command line or in a makefile if it makes use of the Arduino.h Library? , How do I link the Arduino Library to the precompiled chunks I intend to make?

What would you recommend on doing? , What should I do if there is some functionality I'd like to share, but for n amount of reasons I can't release the source code?

pert

So as I understood I need to create a .a file for each of the targets I intend to use. Then create a library that includes those .a files. And finally modifying each of the platform.txt for each of the targets I intend to use.
Correct.

This sounds ok until the last part, because if you update the IDE, you'll need to redo all.
I agree. However, this functionality has already been added to some of the hardware packages, and I think it will eventually be added to the others. It doesn't seem to be a priority for the Arduino developers to add it to Arduino AVR Boards themselves, but they are supportive of it. It's probably just a matter of some member of the community submitting that PR and then waiting a little time until one of the Arduino developers gets the time to review and merge it, then waiting for it to be released in the next version of Arduino AVR Boards. It would have already been done if that original PR didn't have a bug. Similar process process for any other hardware packages that don't already support this feature in their platform.txt files. Also, as I explain in my instructions below, I found that I was able to to use the precompiled feature even with the stock Arduino AVR Boards, so it seems modifications to platform.txt are only needed under specific circumstances.

But there are some steps in the way of creating those .a files that I don't fully understand. How do I compile the library from the command line or in a makefile if it makes use of the Arduino.h Library? , How do I link the Arduino Library to the precompiled chunks I intend to make?
It turns out you can do this very easily using the Arduino IDE, as I'll explain in my instructions.

What would you recommend on doing?
To be honest, I didn't carefully read your post the first time. I kind of skimmed to the part where you asked about precompiling and jumped off there. I was assuming that your problem was something to do with the large number of files causing compilation to take a long time. But now that I read it again, I see you describe the problem like this:
This means there are currently 144 files involved, when building for Arduino I have to manually copy each file from the library directory, to a new directory so all of the files are on the same directory so Arduino ide can find them when compiling. This becomes pretty annoying when debugging code.
I don't understand why you need to copy the files. It seems like you should be able to create  a folder structure that allows you to use the library as-is with the Arduino IDE. You say "so all of the files are on the same directory so Arduino ide can find them when compiling", but the Arduino IDE recursively compiles the contents of the src subfolder of the library when the library is in 1.5 format (library.properties file in the root of the library and source files under the src subfolder).

Since I don't understand your problem, I can't recommend a solution. Maybe if you provide more information then someone here on the forum will be able to give some advice.

What should I do if there is some functionality I'd like to share, but for n amount of reasons I can't release the source code?
I think this is the most common use case for the precompiled feature. I went ahead and did some research on how this works and I can now provide instructions, which I'll do in my next reply.

pert

#6
Dec 16, 2019, 07:07 am Last Edit: Jul 13, 2020, 09:33 am by pert
Open the library.properties file that is in the root folder of your library in a text editor (reference).

Add this line to the file (reference):
Code: [Select]
dot_a_linkage=true

Save the file.

In the Arduino IDE, select a target from the Tools > Board menu.

File > Preferences

Check the box next to "Show verbose output during: compilation".

Click the "OK" button.

Compile any sketch that has an #include directive for your library. This could just be a bare minimum sketch.

After the compilation finishes, examine the contents of the black console window of the Arduino IDE window. The first thing you need to find is the location of the temporary build folder. This should be shown in (among other places) the final command of the output. In my case, the command looks like this:
Code: [Select]
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino5/bin/avr-size" -A "C:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003/PrecompiledTest.ino.elf"
so I know the build folder is at:
C:\Users\per\AppData\Local\Temp\arduino_build_602003

The other thing you need to find in the output is the -mmcu flag passed to the compiler (may be -mcpu instead when compiling for other architectures). In my case it was in this command:
Code: [Select]
"C:\\Users\\per\\AppData\\Local\\Arduino15\\packages\\arduino\\tools\\avr-gcc\\7.3.0-atmel3.6.1-arduino5/bin/avr-gcc" -Wall -Wextra -Os -g -flto -fuse-linker-plugin -Wl,--gc-sections,--relax -mmcu=atmega2560 -o "C:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003/PrecompiledTest.ino.elf" "C:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003\\sketch\\PrecompiledTest.ino.cpp.o" "C:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003\\libraries\\PrecompiledTest\\PrecompiledTest.a" "C:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003/..\\arduino_cache_483105\\core\\core_arduino_avr_mega_cpu_atmega2560_99b4804b5450ac10a663b47fdad40a36.a" "-LC:\\Users\\per\\AppData\\Local\\Temp\\arduino_build_602003" -lm
so you can see the flag was:
Code: [Select]
-mmcu=atmega2560
because I was compiling for the Mega 2560

Copy the .a file from the {temporary build folder}/libraries/{library name} folder to the src/{target MCU} subfolder of your library (reference). In my case, the .a file was at:
C:\Users\per\AppData\Local\Temp\arduino_build_602003\libraries\PrecompiledTest\PrecompiledTest.a
and I copied it to the libraries/PrecompiledTest/src/atmega2560 folder.

Rename the .a file, appending "lib" to the start of the name (reference). In my case, it was renamed to libPrecompiledTest.a

Repeat the above process for all other targets you want to support.

You can now remove the source files from the library, since they are replaced by the precompiled file.

Open library.properties in a text editor.

Remove the line:
Code: [Select]
dot_a_linkage=true

Add the line:
Code: [Select]
precompiled=true

Save the file


That's all you need to do for your library. As for the platform.txt file, I discovered that in my simple proof of concept test library (which does use Arduino.h), I was able to use the stock Arduino AVR Boards 1.8.2 without any modifications. But this issue report indicates this might not always be so? I think it is only needed if you use the ldflags field in library.properties. In that case, you need to make sure platform.txt contains two things:

recipe.c.combine.pattern must contain {compiler.libraries.ldflags} and it must be in the right position in the recipe, as shown in https://github.com/arduino/ArduinoCore-avr/pull/52/files.

This line, anywhere in platform.txt:
Code: [Select]
compiler.libraries.ldflags=
which sets a default definition of that property for cases when it is not automatically defined by arduino-builder. Otherwise it will result in compilation failures in those situations.


You may also find the ldflags field of library.properties useful. My understanding from reading this, and my test, is that defining ldflags is not needed in the case where you have used the expected filename for the .a file: lib{library name}.a, where {library name} is the value of the library's library.properties name field. In the case where you need to use a different file name, then you can specify this with the ldflags field. For example, if the library name was foo, but the archive file name was libbar.a, then you would need to add this line to library.properties:
Code: [Select]
ldflags=-lbar
note that you don't specify the "lib" or ".a" parts of the filename in the -l flag.

Another thing to note is that precompiled libraries are only supported in Arduino IDE 1.8.6 and newer, so the users of your library won't be able to use it with an outdated version of the Arduino IDE.

cheche_romo

Thanks @pert  for taking the time and explaining the entire process, it is really helpful.

I am working my way to the precompiled stage, I'm making my library with the library specification, when it's compiling again, I'll try the precompiled.


bperrybap

#8
Dec 19, 2019, 08:41 pm Last Edit: Dec 19, 2019, 08:55 pm by bperrybap Reason: added table for git tree vs distribution tree.
While you could go down the path of pre-compiled files, I would not go there.
IMO, it seems to be a bunch of work and likely can't fully solve the issues,
since how/where it is done and how the files are installed varies by environment and build tools being used so it seems to have risk in that this capability may or may not even be possible for all the desired combination of tools & environments.
And it will likely be a total PITA to maintain for the environments, since you may need to also create special installation tools - which again can vary not only by environment, but also by OS platform being used.

IMO, it is a matter of creating a proper directory structure or if necessary using some automation scripting to create library distribution packages for each desired environment.

In other words can't you create a directory structure that "just works" in all the environments?
If not, I'd suggest that you use some automation scripts to create a distribution package for each supported environment.
That way the master git repository source tree where the source is maintained doesn't have to exactly match the source tree needed for each distribution and each distribution can have its own github repository as needed to simplify installation.
A github repository is needed for any library that you want to be supported by the Arduino IDE library manager.

I use automation scripts often. I use them on my Arduino library packages for things like creating distribution package releases and setting up github tags.
It does things like build all the documentation files, in some cases from Doxygen, in other cases it extracts git tag information an patches header files to insert revision information or http links to documentation or github files and even automatically commits updated files and generates semver compliant tags so that the Arduino IDE library manager can "see" the new release.
In some cases it builds an alternate local source tree and then creates a zip package from that alternate tree that is put up on a server for users to download.

I have one Arduino very large lcd library with MANY sub directories to organize things like header files, fonts, device files, config files, documentation, examples, bitmaps etc...
For this library, the git development tree is not the same as the distribution release tree:

Here is the development source tree:
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                           448           7601           2534         114327
C/C++ Header                   234           4853          14129          24685
C++                             21           3526          12039          12713
Arduino Sketch                 105           3162           6940          12410
Javascript                     108            282            219           4804
CSS                              8            548            104           2635
Java                            24            484            358           2559
Bourne Shell                     5             73            254            186
Perl                             1              7              0            105
DOS Batch                        3             28             88             50
make                             2             10             18             34
-------------------------------------------------------------------------------
SUM:                           959          20574          36683         174508
-------------------------------------------------------------------------------



and here is the distribution tree:

-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
HTML                           139            402            878          13866
C/C++ Header                   113           1900           5383          10657
Arduino Sketch                  73           1685           4098           6582
C++                              8           1002           2729           4294
Javascript                      88            132            105           2442
CSS                              4            309             45           1560
Java                             1             35             25            259
Perl                             1              7              0            105
make                             1              6             12             14
Bourne Shell                     1              2             12              3
-------------------------------------------------------------------------------
SUM:                           429           5480          13287          39782
-------------------------------------------------------------------------------


--- bill








bperrybap

What should I do if there is some functionality I'd like to share, but for n amount of reasons I can't release the source code?
For the most part you are out of luck as most of the time it isn't possible.

If that is the real reason for trying to move to pre-compiled code, then you will have to very carefully study the various licenses involved of all the code being used to create the final image (not just your code) because in many if not most cases, it is not possible to have some source be closed do to licensing restrictions of other code being used and/or build tool issues, particularly in the Arduino environment.

For example, even though LGPL will allow creating a work that links in pre-compiled objects that use closed sources, it does require than any LGPL code that is used be updateable by the user.

This means that the user must be able replace/modify/change any LGPL code that is used in the work and rebuild the work.
i.e. he must have enough information and tools to be able to rebuild the f/w image and get the part updated with the new f/w.

It also means that your pre-compiled objects can not contain any LGPL or GPL code.
This is often very difficult to do as the code often uses header files and/or library code from the gcc tool set which is licensed LGPL or even GPL, including GPL 3.0 depending on which components are used.

And then there is the possibility that while you may be able to technically comply with the LGPL licensing terms for your library, it becomes impossible for the end user to ever use the library because the only way to use the library is to link it with other LGPL or GPL based code to create the final image.
This other LGPL or GPL code then creates licensing requirements for ALL the code in the image which includes your library and the end user may not be able to comply with the more stricter licensing terms so he legally can't create a f/w images that uses your library.

An example of this might be that a closed  library that ships pre-compiled objects depends on another library, and that other library is licensed as GPL 3.0
GPL 3.0 does not allow linking against any closed source code.
In this case, while the closed source library distribution may technically be LGPL compliant, it is not GPL 3.0 compliant and therefore the closed source library is not usable since it is depending on another piece of code that is GPL 3.0 and GPL 3.0 does not allow being used with closed source.


There are many things to take into consideration when trying to use closed source in an open source environment.

--- bill

cheche_romo

For the most part you are out of luck as most of the time it isn't possible.

If that is the real reason for trying to move to pre-compiled code, then you will have to very carefully study the various licenses involved of all the code being used to create the final image (not just your code) because in many if not most cases, it is not possible to have some source be closed do to licensing restrictions of other code being used and/or build tool issues, particularly in the Arduino environment.

For example, even though LGPL will allow creating a work that links in pre-compiled objects that use closed sources, it does require than any LGPL code that is used be updateable by the user.

This means that the user must be able replace/modify/change any LGPL code that is used in the work and rebuild the work.
i.e. he must have enough information and tools to be able to rebuild the f/w image and get the part updated with the new f/w.

It also means that your pre-compiled objects can not contain any LGPL or GPL code.
This is often very difficult to do as the code often uses header files and/or library code from the gcc tool set which is licensed LGPL or even GPL, including GPL 3.0 depending on which components are used.

And then there is the possibility that while you may be able to technically comply with the LGPL licensing terms for your library, it becomes impossible for the end user to ever use the library because the only way to use the library is to link it with other LGPL or GPL based code to create the final image.
This other LGPL or GPL code then creates licensing requirements for ALL the code in the image which includes your library and the end user may not be able to comply with the more stricter licensing terms so he legally can't create a f/w images that uses your library.

An example of this might be that a closed  library that ships pre-compiled objects depends on another library, and that other library is licensed as GPL 3.0
GPL 3.0 does not allow linking against any closed source code.
In this case, while the closed source library distribution may technically be LGPL compliant, it is not GPL 3.0 compliant and therefore the closed source library is not usable since it is depending on another piece of code that is GPL 3.0 and GPL 3.0 does not allow being used with closed source.


There are many things to take into consideration when trying to use closed source in an open source environment.

--- bill

This is really interesting Bill, I think I'll have to study those licenses before taking a decission.

Go Up