Comments on Arduino 1.5 Specifications

Arduino team, are you listening?

very unlikely
its August, Italy is closed,

Since May?

Hi,

If you want to go with the classic C++ way (a big chain of #ifdefs), you're allowed to do so just ignore the arch/ folder.
This is not explicitly stated in the documentation, maybe its worth adding it, I don't know.

When I first implemented the new library format I've checked its functionality with two example of the Servo library:

  1. the first example uses the classic chain of ifdefs:
    LibTest/libraries/Servo at master · cmaglie/LibTest · GitHub

  2. the second uses the arch/* folders way:
    LibTest/libraries/Servo2 at master · cmaglie/LibTest · GitHub

Both can be used, I could say that its a matter of taste, personally my favorite is the method 2).

Also note the with the library.properties you can filter the library from being displayed/compiled on architectures that are not supported by the library itself.

Thank you for the explanations.

My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!

Another problem is, the shared header is compulsory for any new architectures.

  • What happens if a new architecture requires a modification of the header?
  • Who is going to maintain this repository?

My fear is, this structure looks like pretty much a walled garden...

Please note there's a third option: as for Energia, the libraries folder is under the main architecture folder.

So there's a hardware/lm4f/libraries/servo folder and a hardware/msp430/libraries/servo folder.

@avenue33

avenue33:
My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!

why not?

if a new architecture needs a different header, this means that the header itself is architecture dependent and should be placed in one arch/xxx folder or, alternatively, the architecture-dependent piece of code needs to be placed around #ifdefs.

C

cmaglie:
@avenue33

avenue33:
My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!

why not?

I have the same problem with the eclipse plugin that is also based on (automatically generated) makefiles.
Basically the code included in the project is "the root folder and all subfolders"
Having method 1 as you describe makes I have 1 folder for each library.
For method 2 I have to do something smart because I do not need all subfolders.
In your servo example for method 1 I only need the folder
./libraries/Servo/src ( Great I don't even need to exclude examples)
To be added to the include and source path.
In method 2 I need to include
./libraries/Servo2/src
To the include path (and probably also to the source path)
and
./libraries/arch/${arch}
to the include and source path.
I think this has following disadvantages:

  1. It is more work for the tool (while C/C++ has already fixed this)
  2. It breaks upwards compatibility for tools.
  3. It is not obvious from the code what is going on.
  4. There is no easy way to notice a hardware is not supported.

I was wondering: have you considered a c/cpp file like this?

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif

Best regards
Jantje

@cmaglie

Thank you for your comments.

But what about the other points?

Please note there's a third option: as for Energia, the libraries folder is under the main architecture folder.

@avenue33

Not sure if this helps but arduino 1.5.3 support two library structures. The new one and the old one. So for awakward cross compatible libraries we can use the old structure and #defines?

Sorry if this is not helpful

What I see is the Arduino framework is widening the gap with the C++ standards.

I hope the Wiring++ framework would be more more canonical and make-frendly.

(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

./libraries/Servo2/src
./libraries/Servo2/arch/${arch}

should solve your problem, is that a real burden?

Using a file like:

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif

doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

  1. It is not obvious from the code what is going on.
  2. There is no easy way to notice a hardware is not supported.

It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).

Personally I don't like to duplicate a library into many different places, because this leads to have only a subset of them updated. I've experienced this with many Arduino libraries: some contributions slightly change the API (=> the common headers) on the AVR side but doesn't care to fix things for the less popular SAM.

BTW, if you like to use the libraries folder as you do with energia, I don't see any reason to not continue to do so.

cmaglie:
It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).

Thank you for taking the time to answer our questions.

How this key library.properties file is going to be maintained?

Let's take the following case:

The library.properties file initially contains

architectures=avr,sam

Now,

  • maker A releases a board based on architecture archA and updates the file accordingly
architectures=avr,sam,archA
  • maker B releases another board based on architecture archB and modifies the file to fit its needs
architectures=avr,sam,archB

Who is going to ensure the file integrity and consolidate all the platforms to obtain

architectures=avr,sam,archA,archB

We might end up with many different versions of the library.properties file!

EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.

BTW this is a process that cannot be enforced by the tool.

cmaglie:
(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

./libraries/Servo2/src

./libraries/Servo2/arch/${arch}




should solve your problem, is that a real burden?

It is not a real burden but it is work that moves from the compiler to the tool. Also from a user perspective removing a library is removing 2 folders. I realize this is not a problem for the Arduino ide but it is one for other tool builders.
Please take into account what I have stated before

Most of the 1.5.X changes are great improvements but I see this library change as over-engineering because of the focus on multiplatform. Many libraries are not multiplatform. I think there are enough c++ ways to solve the multiplatform problems for the libraries that need it; without dependencies on the IDE/make.

cmaglie:
Using a file like:

#ifdef ARDUINO_ARCH_AVR

#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif




doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

I think the file scales just as good as a folder with subfolders. To be honest I don't see the problem.
If more files are needed they should be included in the main file. This because servo.cpp is part of the description and not of the implementation.

I guess that I did not explain my main objection to the current proposal clearly.
My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.

My second objection is that I feel this is overengineering
The setup you are proposing may be good (except for my first objection) if you have a library witch is architecture dependant and non-related people are delivering the source for each architecture.
However as you posted in your last answer in that case someone will have to take ownership to arrange for the code.
Paul who has been working on many cross architecture libraries has states he prefers to work with ifdef in the code.
I call this over engineering as you are basically trying to solve a problem (multiple independent people delivering a implementation for the same declaration for different architectures) that does not exist (as they will need to deliver to a single owner) and is only a very small fraction of the existing libraries.
Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?

I agree you are not directly enforcing this setup to library developers but you are enforcing this to the tool makers. This because there will be a competitor screaming the tool is not 100% compatible even though there is not 1 single library around that uses the proposed setup.

Best regards
Jantje

cmaglie:
EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.

History has shown "Maker C" often abandons their library after about 6 months to a year, after they've completed whatever project motivated writing the code. Even today, there are still many minor Arduino libraries that have "#include <WProgram.h>" for pre-1.0 Arduino.

Libraries published by companies that support a shield, breakout board or other hardware they sell are an exception, but even with those, not long after initial release there's little incentive for those 3rd parties to merge patches or pull requests from Maker A & B. They generally will merge patches from "Maker A" who is also a commercial operation and has a sizeable installed base of boards on the market, but "Maker B" who is a hobbyist selling few if any boards will generally be ignored.

BTW this is a process that cannot be enforced by the tool.

Well, of course not.

But the tool could support a way for Maker A and B to supplement the library with extra info, in a way that won't clobber each other and any revisions from Maker C.

I'll certainly be in the Maker A or B roll regarding many libraries. If Maker C won't accept my patches, I'll fork or patch the library as needed. I'll try to do so in the best way your infrastructure supports.

I'm really glad you're concerned about this stuff. 1.5.x is already a huge improvement over 1.0.x. But in designing this structure, it's pretty clear from the many years of Arduino's history that depending on library authors to actively maintain their code and merge patches from 3rd parties is not very practical.

Paul

Thank you for joining the conversation.

One easy way to make every maker happy and avoid central planning, is to set as many sub-folders as architectures... actually pretty much how it works today.

I really like the way Teensyduino and Energia, to name a few, organise the libraries for multiple platforms.

I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?

Jantje:
I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?

If your request is to remove the "arch" folder, then the answer is no
(btw, it was already done and planned right before I noticed that this discussion was going on).

The basic point is that it's confusing to have multiple versions of the same library.
Like, say, take a library that communicates with an external display, say adafruit, just to name one, makes a library that talks to it, and they also provide an optimized version for AVR that uses hardware SPI (or whatever).
If someone else optimizes the library, for example for pic32, I'd rather have them get adafruit to merge the pic32 optimization into the adafruit library, instead of making a new pic32 display library, and in this case, I feel like having separate folders makes it easier, because adafruit can just grab the arch/pic32/ folder straight from the other person and make them responsible for it as opposed to having everything in one folder or one file, which makes it harder to coordinate. Also, adafruit can improve the common part of the library by touching the code in src/*.cpp folder, providing better graphics algorithms for example, that everyone can benefit.

Using a single folder, or #ifdefs within a single file, makes it harder to provide versions of a library for multiple architectures, not easier.

I repeat, this is not a tech matter, and you will face the same problem, even worst, with a single src/ folder, because it requires more work from the original maintainer to include your code.

That's exactly the point of the arch folder. Or maybe you have something else in mind?

Jantje:
Also from a user perspective removing a library is removing 2 folders.

The user remove just the Servo folder that contains both Servo/arch and Servo/src (and all the others files).

Jantje:
It is not a real burden but it is work that moves from the compiler to the tool. I realize this is not a problem for the Arduino ide but it is one for other tool builders.

We should make things simpler for users (and library authors in this case), and this makes life harder for tool's author (both tools and Arduino IDE developers).
BTW this time is just a little more harder, I'm sure you felt much more pain when you faced the auto #inclusion of libraries, or the sketch preprocessing, no? :slight_smile:

Jantje:
My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.

You can have source code in BOTH arch/ and src/, the difference is that in src/ you will put only architecture-independent code.

For example the Ethernet lib has a common src/ with all the supported protocols, and various SPI implementation that are architecture specific.

The Stepper library is a completely architecture independent library, and have no arch/ folders at all.

Jantje:
Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?

I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.

In any case what's the alternative you're proposing? putting all the architecture in one big file? In this case it seems more complicated to merge your patch (because it actually needs code review) and probably I'll give up.

C
thanks for the answer.
No thanks for ignoring the community (again).
Jantje

Cristian,

Overall, I believe 1.5.4 is a huge improvement over 1.0.5.

For years I've been patching/fixing lots of libraries (often fixing bugs and improving performance, not just porting to Teensy), and since releasing Teensy3 one year ago, I now patch nearly all of the ones that ship with 1.0.5. My aggressive patching of files has worked reasonably well for only 2 reasons: #1 most other 3 party board makers don't put much work into porting/patching libraries, and #2 they tend to fork the entire IDE.

In designing this system, I understand you're facing a lot of trade-offs. From what I know so far, not having actually worked closely with the code yet, it looks pretty good.

If I could ask you to consider just one thing as you continue to develop the software, please keep in mind many library authors abandon their project after a period of several months to a year or two. Long-term, depending on the original author to accept and merge patches is just not realistic.

I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.

That's actually very nice, for the source code, at least from my point of view. :slight_smile:

I'm sure it doesn't sit well with tool authors using Makefiles. In fact, on my long-term todo list is developing a better Makefile. It's also certainly going to need a small collection of binary helper tools to be compatible with 1.5.X. Extra work, but I'm really ok with extra work! I'll only want it to be stable do I don't have to redo all that extra work....

In any case what's the alternative you're proposing?

Well, one sore spot would be modifying the library.properties file. For the code, in theory, all my platform specific stuff goes in its own location. But do I get to create my own .properties file that augments the original? Or do I have to modify the one and only library.properties file for that particular library?

Hopefully the library author doesn't feel merging stuff to that .properties file is too burdensome. But what if the library is no longer maintained? Then I need to patch its library.properties file. If my installer has to change that file, then we're back into a similar situation as 1.0.X, where I'm likely to "install" files that destroy changes made by other 3rd parties.

Then again, this is all still pretty hypothetical. So far, I believe I'm the only 3rd party that's patching any substantial number of libraries without forking the entire IDE. But nonetheless, it's a non-ideal situation that could become a problem if someone else starts patching libraries.