Comments on Arduino 1.5 Specifications

@avenue33

Not sure if this helps but arduino 1.5.3 support two library structures. The new one and the old one. So for awakward cross compatible libraries we can use the old structure and #defines?

Sorry if this is not helpful

What I see is the Arduino framework is widening the gap with the C++ standards.

I hope the Wiring++ framework would be more more canonical and make-frendly.

(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

./libraries/Servo2/src
./libraries/Servo2/arch/${arch}

should solve your problem, is that a real burden?

Using a file like:

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif

doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

  1. It is not obvious from the code what is going on.
  2. There is no easy way to notice a hardware is not supported.

It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).

Personally I don't like to duplicate a library into many different places, because this leads to have only a subset of them updated. I've experienced this with many Arduino libraries: some contributions slightly change the API (=> the common headers) on the AVR side but doesn't care to fix things for the less popular SAM.

BTW, if you like to use the libraries folder as you do with energia, I don't see any reason to not continue to do so.

cmaglie:
It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).

Thank you for taking the time to answer our questions.

How this key library.properties file is going to be maintained?

Let's take the following case:

The library.properties file initially contains

architectures=avr,sam

Now,

  • maker A releases a board based on architecture archA and updates the file accordingly
architectures=avr,sam,archA
  • maker B releases another board based on architecture archB and modifies the file to fit its needs
architectures=avr,sam,archB

Who is going to ensure the file integrity and consolidate all the platforms to obtain

architectures=avr,sam,archA,archB

We might end up with many different versions of the library.properties file!

EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.

BTW this is a process that cannot be enforced by the tool.

cmaglie:
(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

./libraries/Servo2/src

./libraries/Servo2/arch/${arch}




should solve your problem, is that a real burden?

It is not a real burden but it is work that moves from the compiler to the tool. Also from a user perspective removing a library is removing 2 folders. I realize this is not a problem for the Arduino ide but it is one for other tool builders.
Please take into account what I have stated before

Most of the 1.5.X changes are great improvements but I see this library change as over-engineering because of the focus on multiplatform. Many libraries are not multiplatform. I think there are enough c++ ways to solve the multiplatform problems for the libraries that need it; without dependencies on the IDE/make.

cmaglie:
Using a file like:

#ifdef ARDUINO_ARCH_AVR

#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif




doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

I think the file scales just as good as a folder with subfolders. To be honest I don't see the problem.
If more files are needed they should be included in the main file. This because servo.cpp is part of the description and not of the implementation.

I guess that I did not explain my main objection to the current proposal clearly.
My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.

My second objection is that I feel this is overengineering
The setup you are proposing may be good (except for my first objection) if you have a library witch is architecture dependant and non-related people are delivering the source for each architecture.
However as you posted in your last answer in that case someone will have to take ownership to arrange for the code.
Paul who has been working on many cross architecture libraries has states he prefers to work with ifdef in the code.
I call this over engineering as you are basically trying to solve a problem (multiple independent people delivering a implementation for the same declaration for different architectures) that does not exist (as they will need to deliver to a single owner) and is only a very small fraction of the existing libraries.
Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?

I agree you are not directly enforcing this setup to library developers but you are enforcing this to the tool makers. This because there will be a competitor screaming the tool is not 100% compatible even though there is not 1 single library around that uses the proposed setup.

Best regards
Jantje

cmaglie:
EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.

History has shown "Maker C" often abandons their library after about 6 months to a year, after they've completed whatever project motivated writing the code. Even today, there are still many minor Arduino libraries that have "#include <WProgram.h>" for pre-1.0 Arduino.

Libraries published by companies that support a shield, breakout board or other hardware they sell are an exception, but even with those, not long after initial release there's little incentive for those 3rd parties to merge patches or pull requests from Maker A & B. They generally will merge patches from "Maker A" who is also a commercial operation and has a sizeable installed base of boards on the market, but "Maker B" who is a hobbyist selling few if any boards will generally be ignored.

BTW this is a process that cannot be enforced by the tool.

Well, of course not.

But the tool could support a way for Maker A and B to supplement the library with extra info, in a way that won't clobber each other and any revisions from Maker C.

I'll certainly be in the Maker A or B roll regarding many libraries. If Maker C won't accept my patches, I'll fork or patch the library as needed. I'll try to do so in the best way your infrastructure supports.

I'm really glad you're concerned about this stuff. 1.5.x is already a huge improvement over 1.0.x. But in designing this structure, it's pretty clear from the many years of Arduino's history that depending on library authors to actively maintain their code and merge patches from 3rd parties is not very practical.

Paul

Thank you for joining the conversation.

One easy way to make every maker happy and avoid central planning, is to set as many sub-folders as architectures... actually pretty much how it works today.

I really like the way Teensyduino and Energia, to name a few, organise the libraries for multiple platforms.

I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?

Jantje:
I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?

If your request is to remove the "arch" folder, then the answer is no
(btw, it was already done and planned right before I noticed that this discussion was going on).

The basic point is that it's confusing to have multiple versions of the same library.
Like, say, take a library that communicates with an external display, say adafruit, just to name one, makes a library that talks to it, and they also provide an optimized version for AVR that uses hardware SPI (or whatever).
If someone else optimizes the library, for example for pic32, I'd rather have them get adafruit to merge the pic32 optimization into the adafruit library, instead of making a new pic32 display library, and in this case, I feel like having separate folders makes it easier, because adafruit can just grab the arch/pic32/ folder straight from the other person and make them responsible for it as opposed to having everything in one folder or one file, which makes it harder to coordinate. Also, adafruit can improve the common part of the library by touching the code in src/*.cpp folder, providing better graphics algorithms for example, that everyone can benefit.

Using a single folder, or #ifdefs within a single file, makes it harder to provide versions of a library for multiple architectures, not easier.

I repeat, this is not a tech matter, and you will face the same problem, even worst, with a single src/ folder, because it requires more work from the original maintainer to include your code.

That's exactly the point of the arch folder. Or maybe you have something else in mind?

Jantje:
Also from a user perspective removing a library is removing 2 folders.

The user remove just the Servo folder that contains both Servo/arch and Servo/src (and all the others files).

Jantje:
It is not a real burden but it is work that moves from the compiler to the tool. I realize this is not a problem for the Arduino ide but it is one for other tool builders.

We should make things simpler for users (and library authors in this case), and this makes life harder for tool's author (both tools and Arduino IDE developers).
BTW this time is just a little more harder, I'm sure you felt much more pain when you faced the auto #inclusion of libraries, or the sketch preprocessing, no? :slight_smile:

Jantje:
My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.

You can have source code in BOTH arch/ and src/, the difference is that in src/ you will put only architecture-independent code.

For example the Ethernet lib has a common src/ with all the supported protocols, and various SPI implementation that are architecture specific.

The Stepper library is a completely architecture independent library, and have no arch/ folders at all.

Jantje:
Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?

I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.

In any case what's the alternative you're proposing? putting all the architecture in one big file? In this case it seems more complicated to merge your patch (because it actually needs code review) and probably I'll give up.

C
thanks for the answer.
No thanks for ignoring the community (again).
Jantje

Cristian,

Overall, I believe 1.5.4 is a huge improvement over 1.0.5.

For years I've been patching/fixing lots of libraries (often fixing bugs and improving performance, not just porting to Teensy), and since releasing Teensy3 one year ago, I now patch nearly all of the ones that ship with 1.0.5. My aggressive patching of files has worked reasonably well for only 2 reasons: #1 most other 3 party board makers don't put much work into porting/patching libraries, and #2 they tend to fork the entire IDE.

In designing this system, I understand you're facing a lot of trade-offs. From what I know so far, not having actually worked closely with the code yet, it looks pretty good.

If I could ask you to consider just one thing as you continue to develop the software, please keep in mind many library authors abandon their project after a period of several months to a year or two. Long-term, depending on the original author to accept and merge patches is just not realistic.

I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.

That's actually very nice, for the source code, at least from my point of view. :slight_smile:

I'm sure it doesn't sit well with tool authors using Makefiles. In fact, on my long-term todo list is developing a better Makefile. It's also certainly going to need a small collection of binary helper tools to be compatible with 1.5.X. Extra work, but I'm really ok with extra work! I'll only want it to be stable do I don't have to redo all that extra work....

In any case what's the alternative you're proposing?

Well, one sore spot would be modifying the library.properties file. For the code, in theory, all my platform specific stuff goes in its own location. But do I get to create my own .properties file that augments the original? Or do I have to modify the one and only library.properties file for that particular library?

Hopefully the library author doesn't feel merging stuff to that .properties file is too burdensome. But what if the library is no longer maintained? Then I need to patch its library.properties file. If my installer has to change that file, then we're back into a similar situation as 1.0.X, where I'm likely to "install" files that destroy changes made by other 3rd parties.

Then again, this is all still pretty hypothetical. So far, I believe I'm the only 3rd party that's patching any substantial number of libraries without forking the entire IDE. But nonetheless, it's a non-ideal situation that could become a problem if someone else starts patching libraries.

After thinking about how to solve this puzzle I must confess I'm out of options.
I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.
The second option is to have the end user do this.

For the first option I will need lots of time (I'm talking months of work here) which I can and do not want to invest.
For the second option we are talking about a support nightmare which I do not want to handle.

It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.

jantje

ps The Teensy solution I designed is also incompatible with the Arduino 1.5 Specification.

Jantje:
It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.

That might be the same fate for embedXcode, as Arduino is concerned, and for the very same reasons: the new nonsensical library structure.

This should have been avoided, should the Arduino team be more open. I started this thread a long time ago, May 11, 2013, 11:53:48 am. The first answer from a member of the Arduino team only came many months later, on September 09, 2013, 11:27:43 am.

Openness means listening to users and taking their comments into account. Arduino doesn't provide a decent IDE, so the Arduino team should at least consider the Arduino plug-ins for standard IDEs like Eclipse, NetBeans, Visual Studio and Xcode.

Ultimately, the late answer and the closeness are about Arduino having no consideration and no respect to those who, like Jantje, Visual Micro and myself to name a few, develop Arduino plug-ins for full-fledged IDEs and contribute to a larger Arduino audience.

I would like to propose a solution which I think is acceptable for Arduino core team and all tool developers.

In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with

#ifdef ARDUINO_ARCH_AVR

and end with

#endif

As ARDUINO_ARCH_{build.arch} is already defined in platform.txt this is the only thing needed.

I hope this is accepted by the core team
Best regards
Jantje

@Jantje

Another option would consist on duplicating the libraries and copy them into a structure plain GNU GCC Tools can understand and manage.

@ avenue33
I think this is a structure plain GNU GCC Tools understand.
What I plan to do is

  1. use the libraries/[libName] folder as source code (excluding Example). This way all code is known by the tool.
  2. only add libraries/[libName]/src to the include path

I tried this in eclipse and it works.
Why? (taking the servo lib as example)
The tool sees all the source files
-Libraries/Servo/avr/Servo.cpp
-Libraries/Servo/sam/Servo.cpp
Due to the define statement ARDUINO_ARCH_AVR in /Servo/sam/Servo.cpp the output file for /Servo/sam/Servo.cpp will not contain any compiled code.
The compile process of /Servo/avr/Servo.cpp on the other hand will generate code.
So if your toolchain supports directory trees and filenames with the same name at different locations in the directory tree this solution should work for you.
IMHO all C/C++ tools support this.
There are some tricks of the trade that need to be used. For instance having twice a header file with the same name. In the Servo example that is ServoTimers.h
By not adding the lib[architecture] folders to the include path and using a include statement with double quotes the file in the folder of the source file will be used.

As stated this works with eclipse. Can you check with your environment?
Best regards
Jantje

Jantje:
It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.

Believe me, this is the last thing that I want to see happen. So, first, thanks for your patience, I'm sure we can find a solution.

Jantje:
In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with

#ifdef ARDUINO_ARCH_AVR

and end with

#endif

well, this is what we are trying to avoid: repeating the same #ifdef when its not needed.

Jantje:
I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.

What your plugin does when a user adds a library?
I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?

cmaglie:
What your plugin does when a user adds a library?
I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?

The embedXcode plug-in imports all the libraries from the Steckbook/libraries folder and indexes them. Same for the libraries for the application libraries from /Applications/Arduino.app/Contents/Resources/Java/libraries, the variant libraries from /Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/variants and the core libraries from /Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/cores.

When the user wants to use a specific library, he/she just adds the corresponding #include statement.

Because embedXcode manages different paltforms —Arduino, Digispark, chipKIT, LaunchPad, Maple, Teensy, Wiring—, the same process is done with the other platforms for the core, variants and application libraries.

cmaglie:

Jantje:
It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.

Believe me, this is the last thing that I want to see happen. So, first, thanks for your patience, I'm sure we can find a solution.

I'm happy to hear this.

cmaglie:

Jantje:
In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with

#ifdef ARDUINO_ARCH_AVR

and end with

#endif

well, this is what we are trying to avoid: repeating the same #ifdef when its not needed.

I don't see why you use the word repeating. if each implementation has 1 header and 1 source file each #ifdef ARDUINO_ARCH_{build.arch} is only needed once (source code) and to be perfect twice (source code and header so the indexer does not see double declarations)
I do not see why you are trying to avoid this. Each and every header file should have something similar to handle multiple inclusions.
IMHO it is also good that you can see (and get guarantee) that this code is only used for a certain platform.

cmaglie:

Jantje:
I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.

What your plugin does when a user adds a library?

When a library is added the plugin adds a symbolic link to the root of the library folder at the location [project]/libraries/[libraryname].
It also adds a include link for all languages (C and C++ in V2 and C C++ asm in V1).
The end user needs to add the include to the code himself.
Note I do not have a delete library functionality. A user can delete the link but that will not remove the includes.

cmaglie:
I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?

You are right about the variant folder being similar but it is not the same.
Let me first explain how I handle the variant/core
I create 2 symbolic links one with the name [project]/arduino/core and one with [project]/arduino/variant. These link to the appropriate location on disk.
I also create 2 include links to /arduino/core and arduino/variant.
This means that when the user selects another board I may need to update the link location of the link names /arduino/core and arduino/variant.
Note that this solution breaks the symmetry between the "disk situation" and the "eclipse project situation".
The main reason why this works is because arduino only uses

#include "pins_arduino.h"

In other words it relies on the include path to find the file.
When we have a mega; makefile will find it at arduino-1.5.2/hardware/arduino/avr/variants/mega/pins_arduino.h.
The indexer however will find it at [project]/arduino/variant/pins_arduino.h
Understanding this difference is important for the next part.

Suppose I do something similar for a library.
I create 2 link locations [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src.
[project]/libraries/[libname]/lib/arch links to ../libraries/[libname]/lib/[arch]
[project]/libraries/[libname]/arch links to ../libraries/[libname]/src
I add [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src. to the include path.
Apart from the maintenance nightmare (There is no delete lib and I don't know how to make one) and apart from the potential performance impact when changing board I expect this to work in all cases for the makefile but not for the indexer.

One scenario I think of is the following:
I'm building a board that has a "different" architecture than avr but is very much alike. Some libraries need to change but some are 100% compatible. Suppose this architecture is called "jantje"
For each library I support I create the folder jantje under [libname]/lib/ where I place my code.
Assume that the Servo lib I need is 100% compatible with the avr lib. What I will do is create 2 files:
Servo.cpp which looks like this

#include ../avr/Servo.cpp

ServoTimers.h which looks like this

#include ../avr/ServoTimers.h

This will work fine for the makefile but not for the indexer as [project]/libraries/[libname]/avr does not exist.

I hope this explanation is understandable; it is clearly not as simple as embedXcode .
Best regards
Jantje