Comments on Arduino 1.5 Specifications

Paul,
Directly from the 1.5 library spec page in the IDE behavior section:

When the user chooses a board the IDE automatically selects the correct library implementation for the board's architecture. If a library doesn't provide an implementation for the currently selected architecture, the library is not displayed cannot be selected. Examples are shown only if the library has an implementation for the currently selected board architecture.

And in Cristian's first post to this thread:
http://forum.arduino.cc//index.php?topic=165827.msg1384527#msg1384527

Also note the with the library.properties you can filter the library from being displayed/compiled on architectures that are not supported by the library itself.

Form that language I interpret it to be a desired intent/goal to have the IDE filter out
libraries and sketches that are not supported on the selected board.
In fact you see this behavior already today in the 1.5.4. IDE.
Select a DUE board, then click
[File]->[Examples] ->[Scheduler]->MotorKnob
This is an example in the Scheduler library.
Now change the board to "Arduino Uno".
The Scheduler library is now gone.
The reverse happens for the "Bridge" library which is AVR only.
I can go in and modify the architectures value in the .properties
file for the library and the IDE uses the information.
So clearly the intent is that the IDE is going to use this information
to determine what is filtered for the user.

While I think it is a good goal and would be a great capability for the IDE
to support for its Arduino users, the information in the .properties file outlined in
the current 1.5 library spec is simply not capable of providing that capability
across the 3rd party cores.
And that is why I brought up all the real-world examples of how the
current spec falls down.

Regarding multi-arch support, I must agree with Bill, requiring a directory structure that isn't compatible with 1.0.X is obviously going to lead to very slow adoption of this spec. As the author of several libraries and the current maintainer of several more, it's pretty easy to choose between publishing in a format that works on all widely used Arduino IDEs, versus a format that only works on the newest.

My point exactly. And that is also part of the reason for my concern.
I worry that the Arduino Team might resort to a heavy handed approach again
to force its adoption like it did in the 1.0 release.

Cristian, I'm really glad you said this:

First I want to reassure you that if you want to stick to the 1.0 lib format, you can safely do that, we are not going to remove old library support. We already discussed that (in a google issue maybe? I can't remember), so I don't know why you're so concerned about that yet.

I'd like to point out some language near the end of the spec, which might be the source of concern:

At some point in the future, the IDE will drop pre-1.5 library support. There is no specific timeline for this, it will be determined once the new library format has become widespread.

Maybe this part of the spec should be updated, if you're really committed to keeping the old format supported?

I totally agree with Paul here. I saw that language very early on when looking over this 1.5 library stuff
many months ago. That language pretty clear states that there is a goal
to remove support for the pre 1.5 libraries.
The first thing I thought is UH OH... Here we go again.
It's going to pre 1.0 vs 1.0 libraries all over again.

--- bill

Ah, yes, you're right. I overlooked the "IDE behaviour" section. It is right there!

As a maker of a 3rd party boards on a different "architecture", this worries me somewhat. My hope is most libraries will use "architectures=*". But a very likely scenario is "architectures=avr", even when the AVR specific stuff can be emulated (as Teensy 3.0 does for many commonly used AVR-isms). I definitely don't want stuff that "just works" made invisible, only because the original library author only tested on a single platform.

Paul,
What is your understanding of the "architectures=" field?
OR what does architectures=avr really mean?
I brought this up earlier but I think it got lost in the noise.

Suppose the Teensy or maniacbug1284p core is updated to be recognized by Arduino 1.5.4
What then?
I can't resolve in my head what the "architecture" field means given the current 1.5 hardware spec
vs the 1.5 library spec.
The hardware spec talks about vendor/maintainer but the 1.5 library spec
talks about "core-dependencies".
The hardware spec talks about "architectures" which live under a vendor/maintainer directory.
So if the "core-dependencies" in the .properties file refers to a vendor/maintainer core and
its sub architectures, how does a library refer to a specific architecture within a specific vendor/maintainer core?
i.e. how would a library specify that it only runs on the Teensy or maniacbug1284p core?
OR what about a core that works on any AVR architecture processor core regardless
of the vendor/maintainer:
Arduino, Teensy, manicbug1284p, Attiny.

In other words how do 3rd party cores get installed in 1.5 and
then how do libraries specify their dependency on these 3rd party cores or their
sub architectures using the .properties file?

And for products like Teensy that include Teensyduino.
Within Teensyduino, is "teensy" the vendor/Maintainer with sub architectures of avr and arm?
And if so, how does a library .properties file for an "AVR only" library
specify that it can only run on the boards that use the Teensy avr core?

As it is today, it does not appear that the .properties file can handle
3rd party cores dependencies very well.

One possible solution might be to alter the architectures field.
Perhaps something like:
architectures: vendor/architecture (optional-version)

That could potentially even allow something like:
architectures: */avr
To specify any vendor that has an "avr" architecture.

--- bill

When I started to look at the implementation of the library spec in 1.5.4 (I mean look at the sub folders of libraries) I was struck by the tag

"architectures=*".

It goes with what Paul already stated several times about "There is not 1 person maintaining the library".
Basically the properties file is common. So it depends on 1 person to maintain. That one person can decide on which platforms a library can/may run :astonished:.
The same counts for other properties.
What struck me the most was that Arduino mostly works on folder names (like for avr and sam in the hardware/arduino folder) and here decides to go for a "described in file" specification. And that file is under control of Arduino :astonished:.
The reasoning is at least "strange" $).

And then I come to something difficult to explain. I mean there are facts (like It is unnecessary hard to implement in eclipse ) and there are ideas behind a approach (Like it should be easy for the library builders and we don't care about tool builders).
The document explains (though not explicit enough to me) the aim but not the reasoning/ideas behind the "solution(s)".

In this particular case my "idea behind the solution" is very oo oriented.
First of all :There are many things that don't fit the model I'll describe; so I'm sure this model was not in mind when the design was done.
In a oo world (surely C++) there is definition and implementation. so I see a definition which is in the src folder :astonished: and a architectural implementation source code is in the lib/[arch] folder :astonished:. (the :astonished: for the folder names)
That would mean that the implementation references the definition but the spec says

For example, if we try to compile the above library for a board based on AVR architecture, the IDE will automatically compile and include the following folders:

Which I understand as both folders are part of the build path and the include path. Why would I need 2 folders in build and include when 1 is implementation and one is definition?
In this same idea there should be a properties file for each implementation stating which architecture it supports. If there is a conflict the specific setting should overrule any more generic setting as we do with boards.txt and platform.txt.
That solves the "I write a definition but I don't know the implementation" problem but introduces a new one.
What if 2 (or more) library implementations state they can implement your boards architecture?
The most common sense would be that the end user can select the "wanted" library implementation.
This can very easily be implemented by having a define in the properties file. In that case selecting a library implementation means adding a define to the compile path and including the correct header.
This also solves (partly) the forced code duplication problem the current spec has. This because a single folder can hold endless architectures.
I know Arduino has something with ifdef but I feel this is a more robust and easy to implement solution covering more cases as the current spec.
I also realize this is completely incompatible with the current folder names. For instance the current [architecture] folder is prbably better a [implementer] folder. The src folder can not contain code but only definition....

As we are at it.
One of the things I liked about the Arduino libraries (pre 1.5) is that 1 library=1 include in the ino file.
The 1.5.4 includes all the headers in the src folder. for 1 library (I don't remember its name) there is even no file with the library name. I think this will confuse the newbe (and puzzle even the more experienced like me).
My preference goes to 1 include in the ino file with the library name (The name of the folder and maybe preceding folders) for 1 library.

Jantje

bill
I guess you are asking the wrong person on how it is currently implemented.
The only more or less 1.5 compatible teensy right now has been build by me. (off course with help of Paul)
As far as I remember (I don't use the ide) it works in the IDE for Teensy2 and Teensy2++ but surely not for Teensy3.
It is still setup in the pre 1.5.3 format (that is without the library specification we are discussing here) so all libraries are under hardware/teensy. Also all executable code is there. So I used teensy as vendor. (pretty obvious)

Paul preferred (like I do) to only have 1 "hardware folder" so I named it all. So in my implementation teensy hardware is called all.
In the oo spirit (I also elaborated about in my previous post) my plugin allows boards.txt to overwrite all!!!!!!! settings in platform.txt.
This allows me to overwrite even the compiler command. Which is needed for Teensy3.
However Teensy3 does not make a library so I had to add a special flag which is not supported by the arduino IDE.

On the hardware/[arch] folder
I never really understood the need for an [architecture] folder. I mean
When a artist buys a due because he has been told he needs that board. Does he care it is sam or avr when he doesn't even know what that means? (I don't know what sam and avr means)
I think it is not compliant with following points in the library guide line

Be kind to the end user.
Match your API to the underlying capabilities. You don’t want to expose implementation details to the user
Organize your public functions around the data and functionality that the user wants.
Use full, everyday words.

But it is not a library? or is it? ]:smiley:

Jantje

Jantje:
bill
I guess you are asking the wrong person on how it is currently implemented.
What I was trying to solicit is to see is if everybody else is seeing the flaws
in the current .properties specification with respect to trying to specify 3rd party cores
or architectures within those 3rd party cores.

The only more or less 1.5 compatible teensy right now has been build by me. (off course with help of Paul)
As far as I remember (I don't use the ide) it works in the IDE for Teensy2 and Teensy2++ but surely not for Teensy3.
It is still setup in the pre 1.5.3 format (that is without the library specification we are discussing here) so all libraries are under hardware/teensy. Also all executable code is there. So I used teensy as vendor. (pretty obvious)

I did similar for chipKIT a year ago.
I have a working chipKIT environment that works with the Arduino 1.5.1 IDE GUI.
It wasn't that big of deal since the libraries were all separate at that time.
This allowed chipKIT to be using 0023 while the avr and sam used 1.5.1

Paul preferred (like I do) to only have 1 "hardware folder" so I named it all. So in my implementation teensy hardware is called all.
In the oo spirit (I also elaborated about in my previous post) my plugin allows boards.txt to overwrite all!!!!!!! settings in platform.txt.
This allows me to overwrite even the compiler command. Which is needed for Teensy3.
However Teensy3 does not make a library so I had to add a special flag which is not supported by the arduino IDE.

I didn't follow that.
You can do pretty much anything to the compiler, linker, and download commands given the
flexibility offered by the platform and board files. I had to completely change things for chipKIT but the flexibility was
there for me to change all the commands around when chipKIT boards were selected without
having to change anything in the Arduino JAVA IDE code.

On the hardware/[arch] folder
I never really understood the need for an [architecture] folder. I mean
When a artist buys a due because he has been told he needs that board. Does he care it is sam or avr when he doesn't even know what that means? (I don't know what sam and avr means)

While agree that end users should have not to know too much low level details,
I didn't follow the rest of your comment about "architectures"
At a minimum, there as to be separate areas for the different toolsets given multiple toolsets
are required to develop across the different processors.
As far as the end user goes, most of what are discussing is completely hidden from
the normal end user.
The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.
But to do that the library writer and core developer has be very familiar with these
differences and how to configure and specify what works together
AND the system must be capable of allowing enough information to be specified
to ensure that the system can select the proper tools and libraries once the user
specifies which board he is using.
In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.
What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.
I also believe that it is more important to resolve the functional issues
before the implementation issues.

What I was trying to solicit is to see is if everybody else is seeing the flaws
in the current .properties specification with respect to trying to specify 3rd party cores
or architectures within those 3rd party cores.

I think we all see some flaws. Not sure we all see the same.
I was also wondering whether anyone represented the ino tool.

I didn't follow that.
You can do pretty much anything to the compiler, linker, and download commands given the
flexibility offered by the platform and board files. I had to completely change things for chipKIT but the flexibility was
there for me to change all the commands around when chipKIT boards were selected without
having to change anything in the Arduino JAVA IDE code.

What I mean is that if platform.txt states
compiler.c.cmd=avr-gcc
and you selected a UNO and the boards.txt file contains
uno.compiler.c.cmd=sam-gcc
the plugin will use sam-gcc while I expect the Arduino IDE to use avr-gcc

I fully agree the boards.txt and platform.txt makes the life of a tools developer for the compilation part easier.
There has already been a discussion on the developers list that the same should be implemented for the upload.

The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.

It is visible in 1.5.2 when you select a board

In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.

I fully agree on this but there is to much focus on Architecture while this could solve many other problems as well.
For instance this NewTone Library - Plug-in replacement for Tone library - Better, smaller, faster - Libraries - Arduino Forum
There is a library for avr for tone and you have a new implementation which is better but takes more space and is still in beta.
Some people will want the old one. other will want the new one. Why not have both and let the user select the most appropriate for him/her?
In this particular case the folder name can not be the architecture as the architecture is the same for the 2.
so you will have tone/arch/stable and tone/arch/beta
the folder /arch/stable contains a property file stating the architectures it runs on and so does arch/beta.
So it can be that /arch/stable supports avr, sam, all and chipKIT but arch/beta only avr.

What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.

Technically yes but it misses on process and ownership.
For example if I created a new tone library I want to add a folder under libraries/tone/arch/ and it should work.
I don't want to be dependent on someone else to change the existing properties file (for instance because this library supports sam implementation which is not yet available). And when the folder is removed the sam implementation is removed as well.
In other words: As a provider of the library I want to be able to specify some fields in properties. The best way to do so is add a properties file which overrules (or adds to) the properties in the parent.
I don't want to be dependent on a tool (which I would need to implement in eclipse) that updates the properties file based on user import and export activity.

Best regards
Jantje

bperrybap:
Paul,
What is your understanding of the "architectures=" field?

I honestly do not really know at this point.

Ultimately, my top priority is making everything "just work" on Teensy. A strong secondary goal is enhancing the Arduino API, ideally in ways compatible with all official Arduino boards, like the advanced ADC API recently discussed on the developer mail list.

I also do want to utilize this multi-platform stuff, but that's a secondary goal. If there's any sort of trade-off, making everything "just work" for customers takes precedence. I already apply lots of patches to the IDE. I want to do things in the simplest, most maintainable ways, but if patching results in more work for me but a better experience for my customers, I'll always optimize for end user experience.

Regardless of what the spec says, I believe what this really ends up meaning will largely depend on how many library authors actually end up using it. I've never been good at predicting the future, so I'm going to resist the urge to make any more predictions. I'm mostly in "wait and see" mode.

Jantje:
What I mean is that if platform.txt states
compiler.c.cmd=avr-gcc
and you selected a UNO and the boards.txt file contains
uno.compiler.c.cmd=sam-gcc
the plugin will use sam-gcc while I expect the Arduino IDE to use avr-gcc

I guess I don't have that expectation and don't see an issue here.
The point of having the user pick a board is to potentially do things that are board (chip) specific,
to allow a means to potentially do something differently for a given board (chip).
If there are different chips supported within the same chip architecture,
you must be able to potentially override any default settings in the architecture (platform.txt) file.
While the above is a very contorted example, the values of the fields in the board file for
a particular board have to be able to override the field values in the platform file.
In my view, what you have described is the way it should and needs to work.
For example, maybe I want to specify a shell wrapper to set up my tools
for source level debugging. So maybe I set up a board type that uses a shell
wrapper script that sets up link maps, DDD, AVaRICE, and my AVR dragon as part
of the build and/or upload process. I might have to override the platform.txt values to
get it to call my wrappers instead of the calling the tools directly.
(I've done this, and did this to get chipKIT up and working on IDE 1.5.1)

The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.

It is visible in 1.5.2 when you select a board

I think perhaps you have misunderstood me.
When there is a mechanism in place that allows a library to specify which boards (chips)
it works on, the user only has to know what board he has. How it works under
the covers is no concern to him.

In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.

I fully agree on this but there is to much focus on Architecture while this could solve many other problems as well.
For instance this NewTone Library - Plug-in replacement for Tone library - Better, smaller, faster - Libraries - Arduino Forum
There is a library for avr for tone and you have a new implementation which is better but takes more space and is still in beta.
Some people will want the old one. other will want the new one. Why not have both and let the user select the most appropriate for him/her?
In this particular case the folder name can not be the architecture as the architecture is the same for the 2.
so you will have tone/arch/stable and tone/arch/beta
the folder /arch/stable contains a property file stating the architectures it runs on and so does arch/beta.
So it can be that /arch/stable supports avr, sam, all and chipKIT but arch/beta only avr.

I think the specific library (NewTone) is a poor example, as it used a different name and different API
that the library it is trying to replace and as such does not conflict with the IDE supplied tone library.
I'll assume you mean that you want to have a new/optional/experimental library that uses the same name and API
as some other library.
That said, I'm not understanding if you are wanting the ability to select which library to use
at the sketch level (sketch A used old and sketch B uses new) or whether you simply
want the ability to let the user configure/install a library than can override an existing library
for a given vendor core & architecture.
I brought up a similar scenario in my very first post of this thread.
It was scenario #2 of this response:
Comments on Arduino 1.5 Specifications - #45 by bperrybap - Suggestions for the Arduino Project - Arduino Forum
Given that the user libraries already override the IDE supplied libraries,
the ability to support this capability should be possible if proper dependencies exist in the .properties files.
i.e. Already today, if you install a library in your user sketchbook/libraries area it will override
a library provided by the IDE when the names are the same.
So if a future build mechanism/system maintains this capability as well as properly picking the appropriate
library to support the chip on the selected board, by looking at the various properties
in the library hierarchy, then it all will "just work".
The key is to properly define the .properties file and how the library hierarchy works.
The hierarchy mechanism should use a directory structure that doesn't require 3rd parties
to have to merge their separate properties information.

What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.

Technically yes but it misses on process and ownership.
For example if I created a new tone library I want to add a folder under libraries/tone/arch/ and it should work.
I don't want to be dependent on someone else to change the existing properties file (for instance because this library supports sam implementation which is not yet available). And when the folder is removed the sam implementation is removed as well.
In other words: As a provider of the library I want to be able to specify some fields in properties. The best way to do so is add a properties file which overrules (or adds to) the properties in the parent.
I don't want to be dependent on a tool (which I would need to implement in eclipse) that updates the properties file based on user import and export activity.
Best regards
Jantje

I totally and completely agree on the need to separate out the property fields and not require
library developers to have to coordinate and merge their properties and directory trees.
As I mentioned earlier, I believe that requiring 3rd parties to have to cooperate
so closely is unlikely to work since it doesn't scale well across different developers.
To me the current 3rd party cooperation/merging requirement
is a one of the biggest short comings of the current 1.5 library specification.

However, it isn't that simple when it comes to using processor "arch" because
what does "arch" really mean when using 3rd party cores?
Again, think 3rd party cores, not just the cores that come with
the Arduino IDE. Toss in Teensy, maniacBug1284p, and attiny cores.
This HUGE ambiguity over architecture is another point I've brought up several times.
According to the 1.5 hardware spec, which is what supposedly defines how new cores are added,
"arch" is a component of the vendor/maintainer core.
Because of this, there isn't necessarily a single core for the "avr" architecture.
So you can't get away with using just architectures names in the .properties file.
For example, while the Arduino teams supplies the Arduino/avr core, it doesn't support all the avr chips.
Teensy will be supplying a partial avr core, and so will maniacbug, and attiny.
A library may only work on the AVR processors that use the Teensy core.
There is no way to specify this.

So again, I believe that it comes back to properly defining the desired functionality first.
In order to do the things like you mentioned above there must first be a requirement to do it,
then there must be way to actually specify it.
Then, there can be discussion over implementation,
which includes the directory structure and the mechanisms to support it.
Some of these are inter-related and intertwined and will need to be considered
at that same time.
However, I think trying to jump directly to the directory tree implementation
before agreeing on the functionality much less the way to specify it, is jumping ahead
too far.

--- bill

I see a certain of confusion about what "architectures" means in the library specification, so let me clarify a bit.

Let's pick the example by bill about maniacBug1284p:

The vendor of this board should proceed this way:

  1. create a 3rd party core for his board:
hardware/maniacbug/avr/platform.txt
hardware/maniacbug/avr/boards.txt
hardware/maniacbug/avr/cores/arduino/[source code for "arduino" core of maniacbug]
  1. create a library that has "platforms=avr" property set.

And he's done.

How the IDE can understand that the library author wants to use the maniacbug core found here:

hardware/maniacbug/avr/cores/arduino/[source code for "arduino" core of maniacbug]

instead of the Arduino core found here?

hardware/arduino/avr/cores/arduino/[source code for "arduino" core of ARDUINO]

simple: because the user has previuosly select the board "Maniac Bug" (or whatever) from the "Tools" menu.
The definitions for this board are inside the boards.txt under maniacbug hardware folder so the IDE knows that it should use the "arduino" core inside that folder.

The concept of "vendor" is there for this reason, we can have multiple vendors that implements the same architecture!

Cristian, if you're still reading, here are some questions:

All 22 libraries that ship in Arduino 1.5.4's libraries folder have "core-dependencies=arduino (>=1.5.0)". What is my Teensyduino installer supposed to do with the core-dependencies field? If "arduino (>=1.5.0)" means it will only work with a core named "arduino", do I need to rename my core from "teensy" to "arduino"? Or do I modify the library.properties files to add the name of my core to the properties files?

Likewise, how am I supposed to install Teensy 3.0's non-avr architecture EEPROM library? Suppose I name Teensy 3.0's architecture "kinetis". It seems logical that I would add a "kinetis" directory inside libraries/EEPROM/arch and copy my architecture EEPROM.cpp there? But what do I do about EEPROM.h? It's currently in libraries/EEPROM/src, shared by all architectures. My EEPROM library customizes that header, adding a constructor which I use to initialize the hardware (unlike AVR, the eeprom hardware needs to be initialized). It looks like I'll need to overwrite the platform neutral header with my customized version. Then, of course there's the library.properties file in EEPROM, which currently says the EEPROM library is only compatible with AVR architecture. Will I need to modify EEPROM's library.properties, for both architectures and core-dependencies?

SoftwareSerial is another library I've ported to Teensy 3.0. Fortunately, SoftwareSerial seems to have a header in libraries/SoftwareSerial/src that includes the architecture specific header, but it looks like I might need to patch library.properties, since both architectures and core-dependencies say it's incompatible with my hardware (even if I copy the code into a new directory to make it compatible). Another complexity with SoftwareSerial is I'll want to patch the examples, which mention the supported pin numbers only for Arduino boards. I'll want to add similar comments for my 3rd boards. How am I supposed to do that?

I recall earlier you mentioned the intended path is to contact the library maintainer and get them to update their library.properties file and add the code. Is that really what you intend for the 22 libraries that ship with Arduino? Generally Arduino has maintained a policy of facilitating 3rd party hardware, but not shipping any code or file that directly support any non-Arduino boards. Arduino's lengthy release cycle could also be, well, frustrating if you're the path to get 3rd party support into those library.properties files.

I did notice 1.5.4 has two old style copies of the Wire library, in hardware/arduino/avr/libraries/Wire and hardware/arduino/sam/libraries/Wire. Maybe this the path? If I have a library like EEPROM ported to another architecture, but it's already present with library.properties incompatible with my hardware, can I put a pre-1.5 format library in my platform/arch/libraries directory? Will it override the same-named copy in the main libraries directory? I was under the impression those directories were the pre-1.5.3 way which was intended to be replaced by library.properties. Is that right? If those are the official way for 3rd party boards to be compatible with the 22 "standard" libraries that ship with Arduino, it really ough to at least be mentioned in the library spec.

When I start supporting 1.5.X, I really do want to do it properly. I honestly do not see what the proper way is to resolve these many issues, at least not without overwriting many library files. Please let me know your thoughts?

Paul thanks for trying it, real examples helps a lot my understanding of the issues.

Well, actually those "*-dependency" fields are not used, and needs to be better defined, so as a temporary workardound you can leave it as is.
Theoretically, "core-dependencies" refers to the core inside your "vendor" folder.
Just to be clear, I imagine (and hope :)) you're doing something like:

hardware/pjrc/kinetis/platform.txt
hardware/pjrc/kinetis/boards.txt
hardware/pjrc/kinetis/cores/arduino/....
hardware/pjrc/kinetis/variants/.....
hardware/pjrc/avr/....same stuff here...

the "core-dependencies: arduino" means that you want to use the "arduino" folder inside the "cores" folder of whatever architecture you are using. So unless you named differently the following folder:

hardware/pjrc/kinetis/cores/arduino/....

you can leave "arduino" in the core-dependency file.

Likewise, how am I supposed to install Teensy 3.0's non-avr architecture EEPROM library? Suppose I name Teensy 3.0's architecture "kinetis". It seems logical that I would add a "kinetis" directory inside libraries/EEPROM/arch and copy my architecture EEPROM.cpp there? But what do I do about EEPROM.h? It's currently in libraries/EEPROM/src, shared by all architectures. My EEPROM library customizes that header, adding a constructor which I use to initialize the hardware (unlike AVR, the eeprom hardware needs to be initialized). It looks like I'll need to overwrite the platform neutral header with my customized version. Then, of course there's the library.properties file in EEPROM, which currently says the EEPROM library is only compatible with AVR architecture. Will I need to modify EEPROM's library.properties, for both architectures and core-dependencies?

We should update the EEPROM.h and add a constructor. It's a simple fix this time, if you send a pull request I can quickly merge it.
(it would be even better to add a begin() method to avoid the C++ constructors fiasco but... anyway...).
You should append ",kinetis" to the platforms=.... field. It's still not the intended way to work (I mean patching a library with an installer) but I see your concerns about IDE release cycles.
I wanted the platform= property but this example let me think about that. I recall that you already pointed this out, maybe on my first request for comments? Do you a have a solution for that?

SoftwareSerial is another library I've ported to Teensy 3.0. Fortunately, SoftwareSerial seems to have a header in libraries/SoftwareSerial/src that includes the architecture specific header, but it looks like I might need to patch library.properties, since both architectures and core-dependencies say it's incompatible with my hardware (even if I copy the code into a new directory to make it compatible).

Same for EEPROM.

Another complexity with SoftwareSerial is I'll want to patch the examples, which mention the supported pin numbers only for Arduino boards. I'll want to add similar comments for my 3rd boards. How am I supposed to do that?

Ok, as said earlier, this is an uncovered part of the specification. Suggestions welcome.

I recall earlier you mentioned the intended path is to contact the library maintainer and get them to update their library.properties file and add the code. Is that really what you intend for the 22 libraries that ship with Arduino? Generally Arduino has maintained a policy of facilitating 3rd party hardware, but not shipping any code or file that directly support any non-Arduino boards. Arduino's lengthy release cycle could also be, well, frustrating if you're the path to get 3rd party support into those library.properties files.

We need to split libraries from the Arduino IDE and put them in their own repositories, its already in the Arduino plans, but these steps, even if it seems easy, requires time to be done properly. The final goal is to push authors to collaborate, and to make, say, a good SoftwareSerial library that is well maintained with a consistent API across platforms.

Of course, this is my thought, in my ideal world. Instead I see, also from comments in this thread, that current trend is that every author starts its own fork of every library, eventually changing the API, or not following the updates that the upstream library adopt in the meantime.

I know also that is frustrating to wait months for a patch to go in. And I perfectly understand your concerns.

I did notice 1.5.4 has two old style copies of the Wire library, in hardware/arduino/avr/libraries/Wire and hardware/arduino/sam/libraries/Wire. Maybe this the path? If I have a library like EEPROM ported to another architecture, but it's already present with library.properties incompatible with my hardware, can I put a pre-1.5 format library in my platform/arch/libraries directory? Will it override the same-named copy in the main libraries directory? I was under the impression those directories were the pre-1.5.3 way which was intended to be replaced by library.properties. Is that right? If those are the official way for 3rd party boards to be compatible with the 22 "standard" libraries that ship with Arduino, it really ough to at least be mentioned in the library spec.

I see that the library search priority is another difficult topic.
The current IDE should work, more or less, like this:

Let's call ARCH the architecture of currently selected board, and VENDOR its vendor name.

  1. The IDE discards all 1.5 libraries that haven't ARCH in the platform= property.
  2. From the remaining libraries it choose the first matching library by searching these folders, in order:
    a) SKETCHBOOK/libraries
    b) ARDUINOPATH/hardware/VENDOR/ARCH/libraries
    c) ARUDINOPATH/libraries

So putting a library inside ARDUINOPATH/hardware/VENDOR/ARCH/libraries works. Again, here there is another compromise to deal with: we can reorder the path priority for example b) c) a), but this won't let the user the possibility to override vendor-provided libraries.

Thanks for taking the time to look at these issues.

Regarding the EEPROM.h, I'm afraid I oversimplified things a bit. I'm also using inline functions that call C code from within the class definition. I don't actually have any C++ code for EEPROM on Teensy 3.0, except in that header. The rest is all C-only, for a number of other reasons I won't mention, for the sake of keeping this reply shorter.

I really do appreciate your offer to merge a pull request quickly. But with 1.5.5 only days from release, I would really prefer to target 1.5.6 and allow more time to carefully consider these complex issues. The last thing I want to do is hastily prepare a pull request and cause breakage in an Arduino release, even if it is labeled beta.

It's tempting to write more regarding the parts of this spec that give my the biggest cause for concern, "core-dependencies=" and "architecture=", but at this moment I'd really prefer to think about these a while longer before more comments. Likewise for customizing the examples.

The library search order is one place where I'd like to make a proposal.

The current IDE should work, more or less, like this:

Let's call ARCH the architecture of currently selected board, and VENDOR its vendor name.

  1. The IDE discards all 1.5 libraries that haven't ARCH in the platform= property.
  2. From the remaining libraries it choose the first matching library by searching these folders, in order:
    a) SKETCHBOOK/libraries
    b) ARDUINOPATH/hardware/VENDOR/ARCH/libraries
    c) ARUDINOPATH/libraries

So putting a library inside ARDUINOPATH/hardware/VENDOR/ARCH/libraries works. Again, here there is another compromise to deal with: we can reorder the path priority for example b) c) a), but this won't let the user the possibility to override vendor-provided libraries.

I agree, a) b) c) is the best search order.

However, I would like to propose a smart strategy for the a) to b) transition. If the user installs a library in their sketchbook, perhaps with "architecture=avr" or any other library.properties line that doesn't match the board they're selected, then b) and c) should be tried to find a better match. Libraries found in b) should always match, and c) is the last location, so really the smart decisions are only truly needed at step a).

Cristian,
You did answer one thing that I thought was probably the case:
The "architecture" names are global across all vendors.

As far as my question goes about how a 3rd party library can specify which core to use
I think you have missed part of the issue.
What I was questioning was more how does a library specify that it
only works with certain specific cores?
For example, suppose the library only works on a specific vendors core
and none of the others - even if they are the same architecture.
i.e. maybe something only works on the pjrc teensy avr core.

Then there is the case where a library only works on a certain chip.
Maybe it only works on a board that has a 32u4 on it
or maybe it only works on board with a m168 or m328.
Libraries only working on specific chips is quite common.
This common real-world scenario can't be handled by the current .properties spec.

What I'm saying is that I wasn't concerned that the spec could get
a library to work when the user selected a board that supported it even when it was
on a 3rd party core or 3rd party board.
It was more that the there is not enough information to prevent the user from trying to use
it on boards that can not support it.
It was my assumption that this was a goal of the .properties stuff in the first place.
To provide enough information to allow the IDE to filter out and not show
libraries and examples that will not work on the board the user has selected.
And if we going to allow the user to pick libraries and examples that won't work
on and/or error off during compilation for the selected board,
we are missing out on what could be done because that is essentially what we have today.

And then what about sketches?
Sometimes users go off and download sketches.
Currently there is no way for a sketch author to specify which cores or libraries may
be required other than with comments in the code.
We see many users posting questions in the forums asking why the sketch or example
that they download isn't compiling because of missing libraries or it not working
on the board that they have.

I guess what bothers me is that the current spec looks like it is only capable of
solving some of the issues while ignoring several current real world issues, and I don't really like partial solutions.


With respect to "cores" and core-dependencies,
I think I now see the problem.
The Arduino team point of view seems to be that the environment as fairly consistent
for a given release.
And for them this is case because they ship a full blob that is all done at the same time and is all the same
level/version. The Arduino team really doesn't have to deal with past releases in their libraries.
However, once you enter into the 3rd party world, this is no longer the case.
In the 3rd party world, a core or library is done and while the Arduino IDE and its cores advance
the 3rd party library and core may not be updated right away or at all.
Also, a 3rd party library and core cannot always depend on only using the latest Arduino release
the way the Arduino team can with its libraries.
Many 3rd party libraries today are capable of running on pre 1.x as well as 1.x
and even 1.5 (by using appropriate ifdefs).
And this is why backward compatibility is so important to the 3rd party world,
whereas to the Arduino team they have no need for backward compatibility
because they essentially re-do everything all at once on each of their releases
and get to leave old releases behind.
This is why the 1.0 library change was no big deal to them, while it was a HUGE deal
the 3rd party developers.

So back to why this matters for the "core-dependencies".
After seeing the response to Paul's question it appears that
there really is no vendor/maintainer core dependency in the spec at all.
The "core-dependencies" is actually a version of a architecture core within all the vendors cores.
There appears to be an assumption that all the cores of a given architecture will/must be at
the same version.

I had asked about about what the "core-dependencies" really meant in previous posts.
My assumption had been that it referred to the vendor/maintainer to allow the library
to specify which vendor's core it worked on.
But now it appears that this is really a revision of the "arduino" core within all the architectures.
Note: The naming seems VERY confusing in that there is a "arduino" vendor
and subdirectory in the hardware directory and now we have another "arduino" name
that means something completely different.
What really confuses me is that the library spec says:

core-dependecies - the core on which this library works on (multiple cores can be specified)

So my assumption was "cores" like Teensy vs ManiacBug vs Arduino etc...
But this seems to not be the case.
It appears that the "core-dependencies" values are referring to the revision
of an architecture core.

While I can see the use for the ability of the library to ensure that a particular "arduino" core API
is supported within a given architecture core, my concern is that it appears to be tracking the
IDE revisions because that's how thins are released for the Arduino team code.
This concerns be greatly.
It also does not appear to allow using or requiring different versions for the different architectures.

Consider this case:
Down the road the "arduino" API advances and you have a situation
with multiple maintainers for a library.
For example,
the latest "arduino" API is now 1.5.9
The library previously supported an avr architecture and works on any API back to 1.5.0
Now support is being added for sam but the sam developer decided to use features
from 1.5.9
So now the sam code requires >=1.5.9 but the avr code can run on >=1.5.0
There does not appear to be a way to specify that.

This is why earlier I brought up the possibility of specifying the core version
on the architecture= values.
It allows the library to run with mismatched versions on different architectures
as well

So you could have:
architectures = vendor/architecture/core (version)

This would support most of the capabilities I've been talking about
including the ability to specify individual vendors cores.
(It wouldn't handle the library only working a specify chip).

So for the above example:
architectures=*/avr/arduino (>=1.5.0), */sam/arduno(>=1.5.9)

which is any vendor with "avr" architecture "arduino" core and then
the desired/needed revisions of that core.

For a library that only works on Pauls teensy avr core:

architectures=pjrc/avr/arduino

For a library that only works on maniacbug
architectures=maniacbug1284p/avr/arduino

For a library that works on any "arduino" core >= 1.5
architectures: //arduino (>= 1.5.0)
(any vendor, any architecture, but requires arduino core >= 1.5.0)

To me this seems to make more sense than the current
core-dependencies since I simply can't resolve how that really
works once you have more than one version and multiple architectures.

etc...

or perhaps some sort of hierarchical structure like what is in
the platform and board files to allow the specific architecture of
board preferences override the more general settings.

For sketches, I think it would seem logical to use the same
type of .properties file.
The sketch would have its library dependencies,
and those libraries may have additional dependencies
to form the final dependencies for the sketch.


All that said, I think that there needs to be more thought to come
up with a way that allows a library to specify that it only works on
a specific chip.

Perhaps using the property values like what is in the board and platform
files is the answer.

--- bill

Bill, precise fine-grain compatibility specs would be excellent on a technical level, but would they also work well on a human level?

A scenario that concerns me looks something like this:

1: User A writes code to interface some sensor to Arduino Uno.
2: Forum users B, C, & D request the code, so he packages it into a library and uses a library.properties for Uno. User A posts the code to a forum topic.
3: Leonardo users can't use the library, even though it would work, only because library.properties is too narrowly specific.
4: User C with a Leonardo adds the specs for Leonardo, posts an updated copy later on the same forum topic.
5: User B has an Arduino Yun, which could run the library, except user C only owns a Leonardo board, and didn't add/test support for Yun.
6: Several months later, Arduino Tre is released. The library could run on Tre, but can't, only because the spec is again too narrow.
7: Later, someone wants to run it on Teensy 2.0. I make a modification for Teensy 2.0 and publish a copy in my Teensyduino installer.
8: Yet more users want to run the library on Arduino Micro, Sparkfun's clones, or other derivitives. Maybe they find the original Uno-only code, or the 2nd port, or perhaps my version.

I could go on and on with this list.....

However, my biggest concern is how code gets ported and adapted to different boards. Real users simply try using a library on an unsupported board. They get errors and ask for help on the forum. Hopefully they post the actual error messages. The "copy error" button in 1.5.X is intended to gently guide users to post actual errors. Expert forum users see the error and suggest fixes. Sometimes the original author updates their library. Sometimes abandoned libraries get picked up by others (really, I believe the most beautiful thing about open source).

My point is the "just try it and see what happens" approach, messy and error-prone, is a pretty important mechanism by which the Arduino community collaborates. Many people who know a lot more about human interaction than me have written about the importance of empowering people to try things and accept/learn from setbacks and failures.

Yes, some libraries (notably ones you've written) are done with an expert approach, where one developer pours tremendous effort into cross-platform compatibility and verification. But most libraries aren't developed that way.

An air-tight system that prevents users from trying to compile code on specific boards the author or maintainer hasn't certified as compatible might spare users from compile errors, but it might also have a pretty negative overall impact on community collaboration and sharing.

Paul,
It isn't possible to have it both ways.
Either the system allows specifying which cores/architectures/boards/chips and other libraries
it works on or it doesn't.
I'm ok with either way.
The problem is having a half way in the middle type of approach which is the way
I view the current .properties spec.

I guess it comes down to what is the point/purpose of all
this .properties stuff?
It doesn't seem to be obvious from looking at the spec or looking
at the behavior of the 1.5.4 IDE.

My assumption was that it was to ensure that all needed resources existed
to support a library. Those resources include, other libraries, cores, and hardware.

If it isn't to prevent a sketch/library from being compiled on non supported h/w
and s/w then I think there are MUCH simpler solutions to this library properties stuff.

--- bill

I really wish I would have put more thought into this when I first saw it several months ago.

The more I think about it now, the more concerned I feel it may create artificial barriers to humans collaborating.

Guys,

everyone of you made great points here.
I would ask you to continue the discussion on the developers mailing list, that is where these discussions should take place.

Please give a concise and clear recap of your concerns, and I'm sure that we can come up to a great library specification.

cmaglie:
Guys,

everyone of you made great points here.
I would ask you to continue the discussion on the developers mailing list, that is where these discussions should take place.

Please give a concise and clear recap of your concerns, and I'm sure that we can come up to a great library specification.

Sorry, too late. I've switched for the LaunchPad platform from Texas Instruments: cheaper, better, stronger.

There's the same Processing-based IDE called Energia which is compatible with the Wiring / Arduino framework. So I can reuse all my projects with no or minimal adaptation. Moreover, Energia manages the MSP430 and TM4C platforms in a very clean way.

The LaunchPad Tiva C TM4C123 is a real bargain at US$13 with 256 KB flash, 32 KB RAM, 80 MHz ARM M4 with FPU, 40 pins and an integrated programmer/debugger. The Connected LaunchPad Tiva C Series TM4C129 goes even further: for US$20, 1 MB of flash, 256 KB of RAM, 120 MHz ARM M4 with FPU, 98 pins, built-in Ethernet and an integrated programmer/debugger.

There's the same vibrant hobbyists community (43oh and Stellarisiti forums) but also professional support at the E2E engineer-to-engineer forum hosted by Texas Instruments.

Of course, I'm using embedXcode :slight_smile: