Arduino Forum

Development => Suggestions for the Arduino Project => Topic started by: avenue33 on May 11, 2013, 12:53 pm

Title: Comments on Arduino 1.5 Specifications
Post by: avenue33 on May 11, 2013, 12:53 pm
Arduino has released two entries of its wiki about the specifications for 1.5:



I like the new structure described in 3rd party Hardware specification. It is cleaner and more open to 3rd party. I've already implemented it in my embedXcode (http://embedxcode.weebly.com) project.

Now, IMHO, I'm not a big fan of the new library specification, as the files of a single library are going to be split among different folders.

See the example provided for Servo:



This means parsing the directories in search for all the files and thus makes the makefiles more complex.

What do you think?

As Arduino is taking the lead over Wiring, I guess all the Processing-based Wiring-derived Arduino-like IDEs have little choice but to comply with the new standard.

I hope Arduino is considering taking a more advanced editor with now standard features as code completion, syntax helper, and a much needed debugger.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on May 11, 2013, 01:34 pm

Now, IMHO, I'm not a big fan of the new library specification, as the files of a single library are going to be split among different folders.

See the example provided for Servo:

  • Servo.h goes under Servo/src/

  • but Servo.cpp is under Servo/arch/avr/.


This means parsing the directories in search for all the files and thus makes the makefiles more complex.

What do you think?

I'm happy you bring this up  ]:D

As provider of the Arduino eclipse plug-in I fully agree. Most of the 1.5.X changes are great improvements but I see this library change as over-engineering because of the focus on multiplatform. Many libraries are not multiplatform. I think there are enough c++ ways to solve the multiplatform problems for the libraries that need it; without dependencies on the IDE/make.
Note that I do like the dependency tree :D

Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on May 11, 2013, 11:48 pm
Duplicate code for each platform will very likely lead to libraries that poorly support the less-used platforms.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on May 12, 2013, 05:52 pm
@Jantje

I'm glad you share my humble opinion.

@Paul

I don't see clearly the link between the availability of a library for a given platform and the new structure for the libraries on the Arduino 1.5 specifications.

Some boards --as Teensy and Digispark-- add plug-ins to the main Arduino IDE. How to proceed with the installation of the libraries included on a plug-in? What will happen when a plug-in provides a header specific to its platform when the header is already on the top folder --like the Servo example with servo.h?

Dealing with conditional pre-processing statement to deal with the different platforms / architectures is a nightmare I've been dealing with when I've developed the embedXcode (http://embedxcode.weebly.com/) template for Xcode. The template currently supports Arduino, chipKIT MPIDE, Digispark, LaunchPad Energia, Maple, Teensy and Wiring.

I rather prefer today's de facto approach with one folder per platform / architecture, with everything related to it: cores, variants, firmwares, libraries, system, bootloader.

See below the examples of
• Arduino 1.5.3 with avr and sam
• Energia 0101E009 (http://energia.nu) with msp430 and lm4f, a fork of Arduino for Texas Instruments' LaunchPads

I'd appreciate some feedback from the Arduino team.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on May 28, 2013, 05:57 am

I'd appreciate some feedback from the Arduino team.


Arduino team, are you there?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pYro_65 on May 28, 2013, 07:06 am

Duplicate code for each platform will very likely lead to libraries that poorly support the less-used platforms.


I like the idea that the 1.5.1 had ( not used later ) where it has a global 'libraries' folder, and then an architecture specific 'libraries' folder. This approach is far more standard, as you could simply define a bunch of macros specifying the Due's existence, allowing me to simply use a ifndef to exclude/include specifics.

Or I could place my entire library in an AVR specifics folder if it was not Due compatible.

Arduino needs to move closer to C++, rather than try and hide it behind the scenes. All it does for newbies is add to the confusion as they do not realise they are using C++, and parts of the standard are not applicable because of Arduino's hidden background changes and inconsistent use of file structures.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on May 28, 2013, 07:39 am

Arduino needs to move closer to C++, rather than try and hide it behind the scenes. All it does for newbies is add to the confusion as they do not realise they are using C++, and parts of the standard are not applicable because of Arduino's hidden background changes and inconsistent use of file structures.


I fully agree with Arduino needing to go closer to C++. That's the very paradigm behind embedXcode (http://embedxcode.weebly.com), the template I've developed for Xcode: plain C++ code and makefiles.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Jun 13, 2013, 11:38 am
Bump!
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Aug 13, 2013, 06:38 pm
Arduino team, are you listening?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: drjiohnsmith on Aug 13, 2013, 07:14 pm
very unlikely
  its August, Italy is closed,

Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Aug 13, 2013, 07:24 pm
Since May?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Sep 09, 2013, 12:27 pm
Hi,

If you want to go with the classic C++ way (a big chain of #ifdefs), you're allowed to do so just ignore the arch/ folder.
This is not explicitly stated in the documentation, maybe its worth adding it, I don't know.

When I first implemented the new library format I've checked its functionality with two example of the Servo library:

1) the first example uses the classic chain of ifdefs:
https://github.com/cmaglie/LibTest/tree/master/libraries/Servo

2) the second uses the arch/* folders way:
https://github.com/cmaglie/LibTest/tree/master/libraries/Servo2

Both can be used, I could say that its a matter of taste, personally my favorite is the method 2).

Also note the with the library.properties you can filter the library from being displayed/compiled on architectures that are not supported by the library itself.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Sep 09, 2013, 01:08 pm
Thank you for the explanations.

My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!

Another problem is, the shared header is compulsory for any new architectures.



My fear is, this structure looks like pretty much a walled garden...

Please note there's a third option: as for Energia, the libraries folder is under the main architecture folder.

So there's a [font=Courrier]hardware/lm4f/libraries/servo[/font] folder and a [font=Courrier]hardware/msp430/libraries/servo[/font] folder.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Sep 09, 2013, 01:26 pm
@avenue33


My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!


why not?

if a new architecture needs a different header, this means that the header itself is architecture dependent and should be placed in one arch/xxx folder or, alternatively, the architecture-dependent piece of code needs to be placed around #ifdefs.

C
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Sep 09, 2013, 02:45 pm

@avenue33


My only concerns is the header is on one folder while the code is on another.

Not easy to deal with using plain makefile!


why not?


I have the same problem with the eclipse plugin that is also based on (automatically generated) makefiles.
Basically the code included in the project is "the root folder and all subfolders"
Having method 1 as you describe makes I have 1 folder for each library.
For method 2 I have to do something smart because I do not need all subfolders.
In your servo example for method 1 I only need the folder
./libraries/Servo/src ( Great I don't even need to exclude examples)
To be added to the include and source path.
In method 2 I need to include
./libraries/Servo2/src
To the include path (and probably also to the source path)
and
./libraries/arch/${arch}
to the include and source path.
I think this has following disadvantages:
1) It is more work for the tool (while C/C++ has already fixed this)
2) It breaks upwards compatibility for tools.
3) It is not obvious from the code what is going on.
4) There is no easy way to notice a hardware is not supported.

I was wondering: have you considered a c/cpp file like this?
Code: [Select]

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif


Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Sep 10, 2013, 04:24 pm
@cmaglie

Thank you for your comments.

But what about the other points?

Quote
Please note there's a third option: as for Energia, the libraries folder is under the main architecture folder.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: visualmicro on Sep 10, 2013, 04:38 pm
@avenue33

Not sure if this helps but arduino 1.5.3 support two library structures. The new one and the old one. So for awakward cross compatible libraries we can use the old structure and #defines?

Sorry if this is not helpful
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Sep 10, 2013, 05:31 pm
What I see is the Arduino framework is widening the gap with the C++ standards.

I hope the Wiring++ framework would be more more canonical and make-frendly.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Sep 11, 2013, 12:35 pm
(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

Code: [Select]

./libraries/Servo2/src
./libraries/Servo2/arch/${arch}


should solve your problem, is that a real burden?

Using a file like:

Code: [Select]

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif


doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

Quote

3) It is not obvious from the code what is going on.
4) There is no easy way to notice a hardware is not supported.


It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).

Personally I don't like to duplicate a library into many different places, because this leads to have only a subset of them updated. I've experienced this with many Arduino libraries: some contributions slightly change the API (=> the common headers) on the AVR side but doesn't care to fix things for the less popular SAM.

BTW, if you like to use the libraries folder as you do with energia, I don't see any reason to not continue to do so.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Sep 11, 2013, 01:20 pm
It's written into "platforms" field of library.properties, IMHO is much more clearer than digging into the code

@avenue33

The goal is to allow to compile the same sketch (without changes) across multiple architectures so, to allow this, there should be a common header and multiple implementations (one for each architecture).


Thank you for taking the time to answer our questions.

How this key library.properties file is going to be maintained?

Let's take the following case:

The library.properties file initially contains
Code: [Select]
architectures=avr,sam

Now,

Code: [Select]
architectures=avr,sam,archA


Code: [Select]
architectures=avr,sam,archB

Who is going to ensure the file integrity and consolidate all the platforms to obtain
Code: [Select]
architectures=avr,sam,archA,archB

We might end up with many different versions of the library.properties file!
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Sep 11, 2013, 01:28 pm
EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.

BTW this is a process that cannot be enforced by the tool.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Sep 11, 2013, 02:23 pm

(sorry for the delay, there days are very busy for me)

@Jantje, you already answered yourself: including the following two paths

Code: [Select]

./libraries/Servo2/src
./libraries/Servo2/arch/${arch}


should solve your problem, is that a real burden?

It is not a real burden but it is work that moves from the compiler to the tool. Also from a user perspective removing a library is removing 2 folders.  I realize this is not a problem for the Arduino ide but it is one for other tool builders.
Please take into account what I have stated before
Quote
Most of the 1.5.X changes are great improvements but I see this library change as over-engineering because of the focus on multiplatform. Many libraries are not multiplatform. I think there are enough c++ ways to solve the multiplatform problems for the libraries that need it; without dependencies on the IDE/make.





Using a file like:

Code: [Select]

#ifdef ARDUINO_ARCH_AVR
#include ../arch/avr/Servo.cpp
#elif ARDUINO_ARCH_SAM
#include ../arch/avr/Servo.cpp
#else
#error The hardware is not supported
#endif


doesn't scale very well, every platform you're going to support need a separate #ifdef section to be added, and you should replicate this structure for every .c / .cpp file contained in your library, this means a quadratic load of maintenance work (N_of_files * N_of_platforms) just to make it compile.

I think the file scales just as good as a folder with subfolders. To be honest I don't see the problem.
If more files are needed they should be included in the main file. This because servo.cpp is part of the description and not of the implementation.

I guess that I did not explain my main objection to the current proposal clearly.
My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.

My second objection is that I feel this is overengineering
The setup you are proposing may be good (except for my first objection) if you have a library witch is architecture dependant and non-related people are delivering the source for each architecture.
However as you posted in your last answer in that case someone will have to take ownership to arrange for the code.
Paul who has been working on many cross architecture libraries has states he prefers to work with ifdef in the code.
I call this over engineering as you are basically trying to solve a problem (multiple independent people delivering a implementation for the same declaration for different architectures) that does not exist (as they will need to deliver to a single owner) and is only a very small fraction of the existing libraries.
Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?


I agree you are not directly enforcing this setup to library developers but you are enforcing this to the tool makers. This because there will be a competitor screaming the tool is not 100% compatible even though there is not 1 single library around that uses the proposed setup.

Best regards
Jantje


Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Sep 12, 2013, 11:46 pm

EDIT: Hopefully, Maker A and B would send their patch to Maker C (the original library author/maintainer), to include it upstream, instead of making their fork.

In this case the work needed by C is to accept the pull request coming from A and B and, eventually, solve the conflict on the "architectures" line of the library.properties.


History has shown "Maker C" often abandons their library after about 6 months to a year, after they've completed whatever project motivated writing the code.  Even today, there are still many minor Arduino libraries that have "#include <WProgram.h>" for pre-1.0 Arduino.

Libraries published by companies that support a shield, breakout board or other hardware they sell are an exception, but even with those, not long after initial release there's little incentive for those 3rd parties to merge patches or pull requests from Maker A & B.  They generally will merge patches from "Maker A" who is also a commercial operation and has a sizeable installed base of boards on the market, but "Maker B" who is a hobbyist selling few if any boards will generally be ignored.

Quote

BTW this is a process that cannot be enforced by the tool.


Well, of course not.

But the tool could support a way for Maker A and B to supplement the library with extra info, in a way that won't clobber each other and any revisions from Maker C.

I'll certainly be in the Maker A or B roll regarding many libraries.  If Maker C won't accept my patches, I'll fork or patch the library as needed.  I'll try to do so in the best way your infrastructure supports.

I'm really glad you're concerned about this stuff.  1.5.x is already a huge improvement over 1.0.x.   But in designing this structure, it's pretty clear from the many years of Arduino's history that depending on library authors to actively maintain their code and merge patches from 3rd parties is not very practical.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Sep 12, 2013, 11:54 pm
Paul

Thank you for joining the conversation.

One easy way to make every maker happy and avoid central planning, is to set as many sub-folders as architectures... actually pretty much how it works today.

I really like the way Teensyduino and Energia, to name a few, organise the libraries for multiple platforms.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Sep 24, 2013, 01:08 am
I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Sep 24, 2013, 12:27 pm


I see this has been implemented in 1.5.4. The sd library uses the src folder and as such is incompatible with my plugin.
I assume our arguments have not been heard.
Can someone of the core Arduino team confirm this?


If your request is to remove the "arch" folder, then the answer is no
(btw, it was already done and planned right before I noticed that this discussion was going on).

The basic point is that it's confusing to have multiple versions of the same library.
Like, say, take a library that communicates with an external display, say adafruit, just to name one, makes a library that talks to it, and they also provide an optimized version for AVR that uses hardware SPI (or whatever).
If someone else optimizes the library, for example for pic32, I'd rather have them get adafruit to merge the pic32 optimization into the adafruit library, instead of making a new pic32 display library, and in this case, I feel like having separate folders makes it easier, because adafruit can just grab the arch/pic32/ folder straight from the other person and make them responsible for it as opposed to having everything in one folder or one file, which makes it harder to coordinate. Also, adafruit can improve the common part of the library by touching the code in src/*.cpp folder, providing better graphics algorithms for example, that everyone can benefit.

Using a single folder, or #ifdefs within a single file, makes it harder to provide versions of a library for multiple architectures, not easier.


Libraries published by companies that support a shield, breakout board or other hardware they sell are an exception, but even with those, not long after initial release there's little incentive for those 3rd parties to merge patches or pull requests from Maker A & B.  They generally will merge patches from "Maker A" who is also a commercial operation and has a sizeable installed base of boards on the market, but "Maker B" who is a hobbyist selling few if any boards will generally be ignored.


I repeat, this is not a tech matter, and you will face the same problem, even worst, with a single src/ folder, because it requires more work from the original maintainer to include your code.


But the tool could support a way for Maker A and B to supplement the library with extra info, in a way that won't clobber each other and any revisions from Maker C.


That's exactly the point of the arch folder. Or maybe you have something else in mind?


Also from a user perspective removing a library is removing 2 folders.


The user remove just the Servo folder that contains both Servo/arch and Servo/src (and all the others files).


It is not a real burden but it is work that moves from the compiler to the tool. I realize this is not a problem for the Arduino ide but it is one for other tool builders.


We should make things simpler for users (and library authors in this case), and this makes life harder for tool's author (both tools and Arduino IDE developers).
BTW this time is just a little more harder, I'm sure you felt much more pain when you faced the auto #inclusion of libraries, or the sketch preprocessing, no? :-)


My main objection is that the source is not under the source folder. The source is actually in a subfolder of the arch folder and the declaration is in the source folder. That is at least for the new architecture dependent libraries.
The architecture independent libraries (and I feel this is the biggest bulk) will have there source under the src folder.


You can have source code in BOTH arch/ and src/, the difference is that in src/ you will put only architecture-independent code.

For example the Ethernet lib has a common src/ with all the supported protocols, and various SPI implementation that are architecture specific.
https://github.com/arduino/Arduino/tree/ide-1.5.x/libraries/Ethernet

The Stepper library is a completely architecture independent library, and have no arch/ folders at all.
https://github.com/arduino/Arduino/tree/ide-1.5.x/libraries/Stepper


Think about this situation: Paul is working on a architecture dependent library and he is doing the teensy part.
I do the avr and sam part. and you are the owner.
Paul delivers 1 file Platformdependentlib.cpp for teensy
I deliver 1 file Platformdependentlib.cpp for avr and sam.
How will you organize this? Will you copy my file to the sam and avr folder? Will you ask me to deliver 2 files?


I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.

In any case what's the alternative you're proposing? putting all the architecture in one big file? In this case it seems more complicated to merge your patch (because it actually needs code review) and probably I'll give up.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Sep 24, 2013, 12:36 pm
C
thanks for the answer.
No thanks for ignoring the community (again).
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Sep 24, 2013, 04:38 pm
Cristian,

Overall, I believe 1.5.4 is a huge improvement over 1.0.5.

For years I've been patching/fixing lots of libraries (often fixing bugs and improving performance, not just porting to Teensy), and since releasing Teensy3 one year ago, I now patch nearly all of the ones that ship with 1.0.5.  My aggressive patching of files has worked reasonably well for only 2 reasons: #1 most other 3 party board makers don't put much work into porting/patching libraries, and #2 they tend to fork the entire IDE.

In designing this system, I understand you're facing a lot of trade-offs.  From what I know so far, not having actually worked closely with the code yet, it looks pretty good.

If I could ask you to consider just one thing as you continue to develop the software, please keep in mind many library authors abandon their project after a period of several months to a year or two.  Long-term, depending on the original author to accept and merge patches is just not realistic.

Quote

I would ask you to provide two file for arch/sam and arch/avr folders, instead of one full of #ifdefs. In this case Paul becomes responsible for arch/kinetis and you for arch/avr and arch/sam. When I receive updates from you, I need to just copy your file straight in their specific folder.


That's actually very nice, for the source code, at least from my point of view.  :)

I'm sure it doesn't sit well with tool authors using Makefiles.  In fact, on my long-term todo list is developing a better Makefile.  It's also certainly going to need a small collection of binary helper tools to be compatible with 1.5.X.  Extra work, but I'm really ok with extra work!  I'll only want it to be stable do I don't have to redo all that extra work....

Quote

In any case what's the alternative you're proposing?


Well, one sore spot would be modifying the library.properties file.  For the code, in theory, all my platform specific stuff goes in its own location.  But do I get to create my own .properties file that augments the original?  Or do I have to modify the one and only library.properties file for that particular library?

Hopefully the library author doesn't feel merging stuff to that .properties file is too burdensome.  But what if the library is no longer maintained?  Then I need to patch its library.properties file.  If my installer has to change that file, then we're back into a similar situation as 1.0.X, where I'm likely to "install" files that destroy changes made by other 3rd parties.

Then again, this is all still pretty hypothetical.  So far, I believe I'm the only 3rd party that's patching any substantial number of libraries without forking the entire IDE.  But nonetheless, it's a non-ideal situation that could become a problem if someone else starts patching libraries.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 11, 2013, 09:50 am
After thinking about how to solve this puzzle I must confess I'm out of options.
I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.
The second option is to have the end user do this.

For the first option I will need lots of time (I'm talking months of work here) which I can and do not want to invest.
For the second option we are talking about a support nightmare which I do not want to handle.

It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.

jantje

ps The Teensy solution I designed is also incompatible with the Arduino 1.5 Specification.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 11, 2013, 11:02 am

It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.


That might be the same fate for embedXcode, as Arduino is concerned, and for the very same reasons: the new nonsensical library structure.

This should have been avoided, should the Arduino team be more open. I started this thread a long time ago, May 11, 2013, 11:53:48 am. The first answer from a member of the Arduino team only came many months later, on September 09, 2013, 11:27:43 am.

Openness means listening to users and taking their comments into account. Arduino doesn't provide a decent IDE, so the Arduino team should at least consider the Arduino plug-ins for standard IDEs like Eclipse, NetBeans, Visual Studio and Xcode.

Ultimately, the late answer and the closeness are about Arduino having no consideration and no respect to those who, like Jantje, Visual Micro and myself to name a few, develop Arduino plug-ins for full-fledged IDEs and contribute to a larger Arduino audience.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 11, 2013, 01:47 pm
I would like to propose a solution which I think is acceptable for Arduino core team and all tool developers.

In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with
Code: [Select]
#ifdef ARDUINO_ARCH_AVR
and end with
Code: [Select]
#endif

As ARDUINO_ARCH_{build.arch} is already defined in platform.txt this is the only thing needed.

I hope this is accepted by the core team
Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 11, 2013, 03:53 pm
@Jantje

Another option would consist on duplicating the libraries and copy them into a structure plain GNU GCC Tools can understand and manage.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 11, 2013, 06:32 pm
@ avenue33
I think this is a structure plain GNU GCC Tools understand.
What I plan to do is
1) use the libraries/[libName] folder as source code (excluding Example). This way all code is known by the tool.
2) only add libraries/[libName]/src to the include path

I tried this in eclipse and it works.
Why? (taking the servo lib as example)
The tool sees all the source files
-Libraries/Servo/avr/Servo.cpp
-Libraries/Servo/sam/Servo.cpp
Due to the define statement ARDUINO_ARCH_AVR in /Servo/sam/Servo.cpp the output file for /Servo/sam/Servo.cpp will not contain any compiled code.
The compile process of /Servo/avr/Servo.cpp on the other hand will generate  code.
So if your toolchain supports directory trees and filenames with the same name at different locations in the directory tree this solution should work for you.
IMHO all C/C++ tools support this.
There are some tricks of the trade that need to be used. For instance having twice a header file with the same name. In the Servo example that is ServoTimers.h
By not adding the lib[architecture] folders to the include path and using a include statement with double quotes the file in the folder of the source file will be used.

As stated this works with eclipse. Can you check with your environment?
Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 15, 2013, 11:14 am

It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.


Believe me, this is the last thing that I want to see happen. So, first, thanks for your patience, I'm sure we can find a solution.


In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with
Code: [Select]
#ifdef ARDUINO_ARCH_AVR
and end with
Code: [Select]
#endif


well, this is what we are trying to avoid: repeating the same #ifdef when its not needed.


I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.


What your plugin does when a user adds a library?
I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?

Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 15, 2013, 12:06 pm

What your plugin does when a user adds a library?
I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?


The embedXcode plug-in imports all the libraries from the [font=Courier]Steckbook/libraries[/font] folder and indexes them. Same for the libraries for the application libraries from  [font=Courier]/Applications/Arduino.app/Contents/Resources/Java/libraries[/font], the variant libraries from [font=Courier]/Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/variants[/font] and the core libraries from [font=Courier]/Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/cores[/font].

When the user wants to use a specific library, he/she just adds the corresponding [font=Courier]#include[/font] statement.

Because embedXcode manages different paltforms --Arduino, Digispark, chipKIT, LaunchPad, Maple, Teensy, Wiring--, the same process is done with the other platforms for the core, variants and application libraries.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 15, 2013, 01:10 pm


It is with pain in the hart that I conclude that I currently see no other option than stopping my Arduino eclipse plugin purely for this library specification.


Believe me, this is the last thing that I want to see happen. So, first, thanks for your patience, I'm sure we can find a solution.

I'm happy to hear this.




In the .cpp and .c files in the library folder
add a #ifdef ARDUINO_ARCH_{build.arch} at the top and #endif at the bottom.
This would mean that all *.c *.cpp files in avr start with
Code: [Select]
#ifdef ARDUINO_ARCH_AVR
and end with
Code: [Select]
#endif


well, this is what we are trying to avoid: repeating the same #ifdef when its not needed.

I don't see why you use the word repeating. if each implementation has 1 header and 1 source file each #ifdef ARDUINO_ARCH_{build.arch} is only needed once (source code) and to be perfect twice (source code and header so the indexer does not see double declarations)
I do not see why you are trying to avoid this. Each and every header file should have something similar to handle multiple inclusions.
IMHO it is also good that you can see (and get guarantee) that this code is only used for a certain platform.





I know how I can modify the Arduino eclipse plugin so the makefile is ok. However this will not work for the indexer. As such the indexer will not know whether you are using sam or avr which reduces the added value of using a real IDE.

There are basically 2 options I still see possible.
The first is to add a lot of code to add and remove path variables when libraries get added or removed.


What your plugin does when a user adds a library?

When a library is added the plugin adds a symbolic link to the root of the library folder at the location [project]/libraries/[libraryname].
It also adds a include link for all languages (C and C++ in V2 and C C++ asm in V1).
The end user needs to add the include to the code himself.
Note I do not have a delete library functionality. A user can delete the link but that will not remove the includes.



I see that you already handle the variant folder: for the libraries it should be the same, or something very similar, what I'm missing?

You are right about the variant folder being similar but it is not the same.
Let me first explain how I handle the variant/core
I create 2 symbolic links one with the name [project]/arduino/core and one with [project]/arduino/variant. These link to the appropriate location on disk.
I also create 2 include links to /arduino/core and arduino/variant.
This means that when the user selects another board I may need to update the link location of the link names  /arduino/core and arduino/variant.
Note that this solution breaks the symmetry between the "disk situation" and the "eclipse project situation".
The main reason why this works is because arduino only uses
Code: [Select]
#include "pins_arduino.h"
In other words it relies on the include path to find the file.
When we have a mega; makefile will find it at arduino-1.5.2/hardware/arduino/avr/variants/mega/pins_arduino.h.
The indexer however will find it at [project]/arduino/variant/pins_arduino.h
Understanding this difference is important for the next part.

Suppose I do something similar for a library.
I create 2 link locations [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src.
[project]/libraries/[libname]/lib/arch links to ../libraries/[libname]/lib/[arch]
[project]/libraries/[libname]/arch links to ../libraries/[libname]/src
I add [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src. to the include path.
Apart from the maintenance nightmare (There is no delete lib and I don't know how to make one) and apart from the potential performance impact when changing board I expect this to work in all cases for the makefile but not for the indexer.

One scenario I think of is the following:
I'm building a board  that has a "different" architecture than avr but is very much alike. Some libraries need to change but some are 100% compatible. Suppose this architecture is called "jantje"
For each library I support I create the folder jantje under [libname]/lib/ where I place my code.
Assume that the Servo lib I need is 100% compatible with the avr lib. What I will do is create 2 files:
Servo.cpp which looks like this
Code: [Select]
#include ../avr/Servo.cpp
ServoTimers.h which looks like this
Code: [Select]
#include ../avr/ServoTimers.h
This will work fine for the makefile but not for the indexer as [project]/libraries/[libname]/avr does not exist.

I hope this explanation is understandable; it is clearly not as simple as embedXcode .
Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 15, 2013, 01:15 pm
avenue33
From your explanation it is not clear to me (and I guess the same for cmaglie) how you select the correct pins_arduino.h.
Can you elaborate on this?
Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 15, 2013, 02:09 pm
There are two different things:


The [font=Courier]pins_arduino.h[/font] specific to the board selected --e.g. Arduino Leonardo-- is managed with the choice of the variants folder...

[font=Courier]/Applications/Arduino.app/Contents/Resources/Java/hardware/arduino/variants/leonardo[/font]

... through the [font=Courier]boards.txt[/font] file...

Code: [Select]
leonardo.build.variant=leonardo

embedXcode considers the sketch as a standard C++ program --actually, a library of main.cpp-- and doesn't perform under-the-hood hidden processes. Everything is managed by a makefile and the GNU GCC tools-chain.

The biggest obstacle for add-ons like Jante's for Eclipse or mine for Xcode lies on those under-the-hood hidden processes done by the Arduino IDE.

Or to put it with other words: the Arduino team should understand that Arduino is actually two things (from a software point of view)


That framework should remain usable with any standard IDE like Eclipse, NetBeans, Visual Studio or Xcode, to name a few. Hence this very thread.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 15, 2013, 02:13 pm
I don't have much to add at the moment, but I'd like to let you know I'm following this discussion.

Sometime next year I'm planning to create a makefile + utility (a native command line program, compiled for all platforms) that more faithfully replicates Arduino's build process.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 15, 2013, 02:25 pm

I don't have much to add at the moment, but I'd like to let you know I'm following this discussion.

Sometime next year I'm planning to create a makefile + utility (a native command line program, compiled for all platforms) that more faithfully replicates Arduino's build process.


Paul, thank you for your support.

Actually, I'm not sure the Arduino way of compiling a sketch is the most adequate way with so many under-the-hood hidden processes...

The Arduino IDE is lacking a critical feature more powerful MCUs --and thus larger projects-- call for: debugging. Only Texas Instruments offers the Arduino-based Energia framework with the debugging tools from Eclipse-based Code Composer Studio 6 (http://processors.wiki.ti.com/index.php/CCSv6).

So why not to focus your makefile + utility on a standard IDE-friendly project?

I know that "developing an Arduino sketch is not programming" but let's stop the fairy tales and face the reality: avoiding any reference to C++ makes portability and scalability very difficult to achieve, when the framework is strong enough for large project.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 15, 2013, 04:22 pm

I don't see why you use the word repeating. if each implementation has 1 header and 1 source file each


this is not true, an implementation can have many files. Look at the TFT or Ethernet library:
https://github.com/arduino/Arduino/tree/ide-1.5.x/libraries/Ethernet
https://github.com/arduino/Arduino/tree/ide-1.5.x/libraries/TFT

Nonetheless there are some maintainers that actually keeps tens of libraries, the time that you save on the development of the tool is time that you put on the back of the library author. With the 1.5 library specification we are focusing to this audience, not the tools developers.

Quote

I do not see why you are trying to avoid this. Each and every header file should have something similar to handle multiple inclusions.
IMHO it is also good that you can see (and get guarantee) that this code is only used for a certain platform.


Because, there are a lot of project that uses the same approach, and they don't need these guards, because IMHO are useless. One for all (but you can find a lot more that adopt same structure):
https://github.com/mirrors/linux/tree/master/arch

Note: kernel uses the makefile and the GNU GCC tools-chain.

If you google "c++ multiplatform" there are a number of forum that suggests the current 1.5 library structure is the "best" way to go:
http://stackoverflow.com/a/3627208/1655275
http://stackoverflow.com/a/14592359/1655275
these are the first two that are relevant but there are many more.

Quote

When a library is added the plugin adds a symbolic link to the root of the library folder at the location [project]/libraries/[libraryname].
It also adds a include link for all languages (C and C++ in V2 and C C++ asm in V1).
The end user needs to add the include to the code himself.
Note I do not have a delete library functionality. A user can delete the link but that will not remove the includes.


Ok, so it links for example:
Project/libraries/Servo -> Arduino-1.0/libraries/Servo
Let's forget about the remove lib functionality for now.

Quote

You are right about the variant folder being similar but it is not the same.
Let me first explain how I handle the variant/core
I create 2 symbolic links one with the name [project]/arduino/core and one with [project]/arduino/variant. These link to the appropriate location on disk.
I also create 2 include links to /arduino/core and arduino/variant.
This means that when the user selects another board I may need to update the link location of the link names  /arduino/core and arduino/variant.


Link Project/arduino/core -> Arduino/hardware/arduino/avr/cores/arduino
Link Project/arduino/variant -> Arduino/hardware/arduino/avr/variant/leonardo (for example)
Add to build/include path: Project/arduino/core
Add to build/include path: Project/arduino/variant

Quote

Note that this solution breaks the symmetry between the "disk situation" and the "eclipse project situation".
The main reason why this works is because arduino only uses
Code: [Select]
#include "pins_arduino.h"
In other words it relies on the include path to find the file.
When we have a mega; makefile will find it at arduino-1.5.2/hardware/arduino/avr/variants/mega/pins_arduino.h.
The indexer however will find it at [project]/arduino/variant/pins_arduino.h
Understanding this difference is important for the next part.


That's perfectly clear for me until now, it reproduce exactly what Arduino IDE does.

Quote

Suppose I do something similar for a library.
I create 2 link locations [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src.
[project]/libraries/[libname]/lib/arch links to ../libraries/[libname]/lib/[arch]
[project]/libraries/[libname]/arch links to ../libraries/[libname]/src
I add [project]/libraries/[libname]/lib/arch and [project]/libraries/[libname]/src. to the include path.
Apart from the maintenance nightmare (There is no delete lib and I don't know how to make one) and apart from the potential performance impact when changing board I expect this to work in all cases for the makefile but not for the indexer.


this is less clear to me, just to clarify, you are saying (supposing a Leonardo board is selected):

Link Project/libraries/Servo/lib/arch -> Arduino-1.5/libraries/Servo/lib/avr
Link Project/libraries/Servo/arch -> Arduino-1.5/libraries/Servo/src
Add to build/include path: Project/libraries/Servo/lib/arch
Add to build/include path: Project/libraries/Servo/arch

but I guess you typo the first link probably you meant:

Link Project/libraries/Servo/lib/arch -> Arduino-1.5/libraries/Servo/arch/avr
Link Project/libraries/Servo/arch -> Arduino-1.5/libraries/Servo/src
Add to build/include path: Project/libraries/Servo/lib/arch
Add to build/include path: Project/libraries/Servo/arch

I still doesn't understand why your links are named lib/arch and arch, instead of arch/avr and src like in the Servo lib, something like that:

Link Project/libraries/Servo/arch/avr -> Arduino-1.5/libraries/Servo/arch/avr
Link Project/libraries/Servo/src -> Arduino-1.5/libraries/Servo/src
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

but besides that still I don't see any problem with this setup.
Why you say that the indexer would fail?

But let's go on.

Quote

One scenario I think of is the following:
I'm building a board  that has a "different" architecture than avr but is very much alike. Some libraries need to change but some are 100% compatible. Suppose this architecture is called "jantje"


Ok

Quote

For each library I support I create the folder jantje under [libname]/lib/ where I place my code.


you create for every lib the [libname]/arch/jantje/..... folder, for example Servo/arch/jantje/....

Quote

Assume that the Servo lib I need is 100% compatible with the avr lib. What I will do is create 2 files:
Servo.cpp which looks like this
Code: [Select]
#include ../avr/Servo.cpp
ServoTimers.h which looks like this
Code: [Select]
#include ../avr/ServoTimers.h


Oh well, I see your point, but since you're providing another architecture (even if is very close to avr), you'll probably do a better job to copy over those file in your private implementation instead of including with the relative path ../avr/.

Proposal: why you don't include the whole library? something like that:

Link Project/libraries/Servo/ -> Arduino-1.5/libraries/Servo/
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

note that you link the whole library but you add to the include path only the relevant piece for the selected architecture.

C

Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 15, 2013, 06:36 pm


Link Project/libraries/Servo/ -> Arduino-1.5/libraries/Servo/
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

note that you link the whole library but you add to the include path only the relevant piece for the selected architecture.


This is what  I do in my plugin and I would like to continue doing. However without the #ifdef in the source code as I propose this won't work. The problem here is that eclipse does not support negations nor environment variables for exclusions. for instance now I exclude /libraries/*/?xamples to exclude library examples from the project.
I can not do something like exclude !(/libraries/*/${architecture})
I can do those exclusions when the makefile is generated but I can not do it for the indexer.

I disagree that the links prove having a #ifdef per file is bad:
http://stackoverflow.com/questions/3627127/writing-cross-platform-c-code-windows-linux-and-mac-osx/3627208#3627208 (http://stackoverflow.com/questions/3627127/writing-cross-platform-c-code-windows-linux-and-mac-osx/3627208#3627208)
Quote
instead of repeating yourself and writing the same #ifdef .... lines again, again, and again, you're maybe better of declaring the probe() method in a header, and providing three different source files, one for each platform. This also has the benefit that if you add a platform you do not have to modify all of your existing sources, but just add new files. Use your build system to select the appropriate source file.

http://stackoverflow.com/questions/14592179/c-cross-platform-development-to-avoid-preprocessor-directives/14592359#14592359 (http://stackoverflow.com/questions/14592179/c-cross-platform-development-to-avoid-preprocessor-directives/14592359#14592359)
Quote
In essence, you are doing it wrong if you have a lot of #ifdef in the code itself.

I'm talking about 1 #ifdef per file. That is not a lot of #ifdef's in the code.

Note I don't object against a file/folder per architecture. I can see the benefit for certain project types.
I do object against code copying as you proposed I should do for my jantje architecture.
I do feel the comparison between arduino multi architecture libraries and linux is not 100% correct.
One of the responses you referred to states
Quote
.. (2) reduces mental effort to read it for someone who's only familiar with only one platform,  ..

The Arduino platform itself is meant to take away the mental effort of switching architectures to the developer. Therefore each developer becomes a multi architecture developer. The Arduino environment does a great job in supporting this work.
Where as a linux development environment is setup with focus to a developer who focusses on 1 architecture only.

You can clearly see the difference in the linux github link you send and the arduino library setup.
As linux is the project the folder for the code for a architecture is located at
[project]/arch/[architecture]/
Where in Arduino they are in
[arduino version]/libraries/[libname]/arch/[architecture]
As the libname is in front of the architecture it shows clearly that architecture is less foreground in Arduino world.
Note that in Arduino 1.5.2 the setup was more  linux like (libraries organized by architecture) and I could support it.

Quote
I still doesn't understand why your links are named lib/arch and arch, instead of arch/avr and src like in the Servo lib, something like that:

Link Project/libraries/Servo/arch/avr -> Arduino-1.5/libraries/Servo/arch/avr
Link Project/libraries/Servo/src -> Arduino-1.5/libraries/Servo/src
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

That is because I have dyslexia.  In other words it would be like you think it should be.
Edit: I would rename avr so I can have a link include. Just like I have variant for the pins_arduino.h

Jantje



Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 15, 2013, 06:46 pm

You can clearly see the difference in the linux github link you send and the arduino library setup.
As linux is the project the folder for the code for a architecture is located at
[project]/arch/[architecture]/
Where in Arduino they are in
[arduino version]/libraries/[libname]/arch/[architecture]
As the libname is in front of the architecture it shows clearly that architecture is less foreground in Arduino world.
Note that in Arduino 1.5.2 the setup was more  linux like (libraries organized by architecture) and I could support it.


Code: [Select]
[project]/arch/[architecture]/

That is exactly the solution I proposed in this very post dated May 12, 2013 (http://forum.arduino.cc//index.php?topic=165827.msg1238260#msg1238260), already used by and Energia and chipKIT MPIDE.

For example, Energia manages two platforms: MSP430 and LM4F, with with one folder per platform / architecture, with everything related to it: cores, variants, firmwares, libraries, system, boot-loader. Same for chipKIT MPIDE.


Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 16, 2013, 10:41 pm


Link Project/libraries/Servo/ -> Arduino-1.5/libraries/Servo/
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

note that you link the whole library but you add to the include path only the relevant piece for the selected architecture.

This is what  I do in my plugin and I would like to continue doing. However without the #ifdef in the source code as I propose this won't work. The problem here is that eclipse does not support negations nor environment variables for exclusions. for instance now I exclude /libraries/*/?xamples to exclude library examples from the project.
I can not do something like exclude !(/libraries/*/${architecture})
I can do those exclusions when the makefile is generated but I can not do it for the indexer.


Ok, so its the eclipse plugin that is not much configurable, I thought that it would index only the folders included in the build path (as reasonably expecting) and not the entire project, but this is not case, good to know.

Then, you should go back to the first solution I proposed and link the two paths:

Link Project/libraries/Servo/arch/avr -> Arduino-1.5/libraries/Servo/arch/avr
Link Project/libraries/Servo/src -> Arduino-1.5/libraries/Servo/src
Add to build/include path: Project/libraries/Servo/arch/avr
Add to build/include path: Project/libraries/Servo/src

(you can name the linked folders at your wish, of course)

This solution involves an acceptable amount of work you wish to dedicate to your plugin?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 22, 2013, 07:52 am
Here are my comments.
My overall comment summary is that this specification does help some issues
but also creates issues, and that the issues it solves are mostly for the
benefit of Arduino team and it's IDE release which is static for
a given release.
The 3rd party world is much more complex and has different wants & needs.
Because of this, the existing .properties format is lacking critical
information fields to be useful for adding 3rd party cores and libraries.
The current format also appears to start to break down when adding 3rd party cores
and libraries.


A big missing concept is to separate the .properties for the library
supplied with the library before it is installed with that of an
IDE private post install library information and directory structure.
i.e. the IDE should take information from a library as it is being
installed (or removed) and use it to create/manage a private "master" "information file"
and directory tree for the library.
As it is today, it is a real mess and requires lots of external merging, updating,
and cooperation by 3rd parties and external tools
to make it work since it appears that IDE (or future library manager)
is trying to take a hands off approach to managing/merging the information during
install/remove of the libraries.
The type of cooperation among 3rd parties necessary to do this type of synchronization
of the libraries .properties file and directory structure
is very unlikely to actually occur.

I will hazard a guess that this new library structure has the potential to blow up
worse than the decision made at the 11th hour on the 1.x release to intentionally break 100%
of the pre 1.x libraries, particularly if the Arduino team eventually takes
a similar hard-line stance on trying to force everyone to adopt it
by removing support for libraries that don't adopt it.

I also observe that it is very interesting the Arduino team has chosen not to
use this library specification and directory structure for its own core
libraries.  In my view, if this specification was good enough, it could
also be used on the core code libraries, because after all, the core code
library is essentially just another library.


In looking closer, I'd like to first look at the functionality offered rather than
some of the deeper issues like include paths and command line compatibility.

The .properties file has a field called: core-dependencies
What isn't clear about this field is exactly what it is.
Is the core-dependencies referring to the vendor/mantainer mentioned on the
1.5 3rd party Hardware spec (HardwareFolders structure)?

To me this makes the most sense since in the architecture folders are directly under
a directory name that seems to match the "core-dependencies" name. ("Arduino")
It would be helpful if the terminology were more consistent since the
library .properties spec and the hardware folders directory structure
are directly related.

Now things start to get complicated really fast.

Core versions are supported in the .properties file.
So where does the "core" version come from?
The 1.5 library spec shows it as being part of the core-dependencies.
Hopefully it is not the IDE version.
The core version really needs to be tied
to the actual architecture core code not to a revision of a GUI shell/wrapper around
the tools.
It should be possible to have different  cores that are at
different versions installed on an IDE.
This will be particularly true when it comes to 3rd party cores.
For example, what if a user wants to load up a 1.0 or maybe even a pre 1.x
pic32 core using the 1.5x IDE?
It should be possible to do that.
To me, the core-dependencies revision is very ambiguous.
The platforms.txt file down under the architecture file
(where it should be because it documents the revision of that core)
has a version in in it.
Is the platforms.txt version used for anything?
So the questions becomes what is the core-dependencies version,
where does it come from, and how is it used?


The .properies spec supports multiple architectures.
How do you match up which core-dependencies goes with which architecture
when there are multiple of each?

Also, if there are multiple cores supported, not every architecture is supported
or even exists under each "core" assuming "core" means vendor/maintainer in
the 3rd party hardware spec sense.

The current .properties spec assumes that all the examples provided
will work with all the combinations of cores and architectures that the
library supports. This may not necessarily be the case.
What if certain examples only run on certain cores or architectures,
or potentially only certain chips within certain architectures of a given core?

There is currently no way to specify which cores or architectures or chips/boards
an example actually works with.




So now some questions on how to actually use it in real life situations.

1) A 3rd party library that only works on a specific AVR chip.
An example of this might be a library that provides some USB capability but
only runs on the 32u4.
What goes in the .properties file to specify that the library only
works on 32u4 based boards?

As it is today, this can't be done. A user will see it on all avr based boards and get
errors on all non 32u4 boards.

BTW, The existing 1.5.4 examles have a related problem, in that there are examples
that error during compling on any board other than leaonardo.
(Seems like point of all this .properties stuff was prevent this kind of stuff)

2) A 3rd party would like to provide an enhanced version of a library
that replaces an IDE provided library but only when a certain (chip) is used.
This is similar to #1 but replaces the library supplied with the IDE
when a board of a given chip is selected.
An example of this might be a softserial replacement for atmega328.

3) 1.x vs 1.5 libraries.
The issue here is that multi-arch 1.5 libraries will not work on 1.x.
This has now doubled the amount of work a 3rd party developer has to do
if they wish to support both.
The simpler solution will be to just support one or the other.
Developers may choose to use 1.x (I know I will) given that option allows
the code to run on both IDEs.
My concern is that the Arduino team will once again take the hard-line
approach and disable the 1.5 IDE from supporting pre 1.5 libraries in
order to force its use.

4) 3rd party libraries for specific cores or boards.
Consider a case where there is a 3rd party library that only works
on a specific core or a specific board.
Perhaps a vendor has created their own 3rd party "duino" board and has provided
a custom library for it.
The current .properties does now allow specifying down to that level.

5) multiple 3rd party libraries with the same name for different cores/chips/boards.
This is possible but in order to support this, the IDE (library manager) needs to be
smart enough to allow these multiple libraries by maintaining the library tree
vs just blindly taking from the library itself.
Say for example, it should be possible to install multiple core/chip/board specific
libraries and have the IDE (library manager) keep track of it so that the proper
library only shows up and is used when the associated board is selected.

6) It appears to make it very difficult to share code with Non-Arduino projects
I have concerns that this structure will make it very difficult to have
multi arch code that is shared between Arduino and non Arduino environments.
I have a glcd library that currently works on m168/338, Leonardo, Mega,
Chipkit boards, Teensy3, mighty1284p core, and several others.
It also has chip specific i/o routines for many different processors that it uses to speed up the i/o
because of the slow digitalxxx() core routines.
I also am in the process of creating a "bare metal" option to allow
it to be used in raw C++ environments, without Arduino.
At this point in time,
I have no intention of converting it over to this 1.5 library spec as it doesn't
make things easier, and would precluded it from running in several of the other
environments.




I guess my message in all this is that my view of the new 1.5 library spec
from functional point is that while it helps out in some cases, it doesn't go nearly
far enough to solve many of the real-world 3rd party issues.

I also believe that from a implementation stand point, it looks very bleak
to it working if a library needs to come from or be supported by different people
or organizations.

Overall, I have many concerns about this specification.


--- bill

Title: Re: Comments on Arduino 1.5 Specifications
Post by: visualmicro on Oct 22, 2013, 01:44 pm
I think it would be a good idea to step back and look at the reasons why the newer structure is required.

Previously it was possible for various library authors to support some architectures but not all. This allowed users to include a library in projects that do not support the selected architecture. Very confusing.

The new structure allows the "Library Import" menu item(s) to accurately show the libraries that are valid for a project/architecture. This removes a whole bunch of support issues.

It might be that this point could have been achieved another way, such as a properties file and the old lib structure however the new structure works really well with many C++ intellisense systems. In fact in Visual Studio, as we  switch between Arduino architectures, we see "red squiggles" for "unsupported" #includes.  So this new system works very well indeed and is very clear for newer users.

By the sound of it even the older structure was a problem for some systems where by all library paths need to be combined into a "master list" of include paths. This is not how the Arduino compiler is normally designed to work, combining all lib paths regardless of the sketch #includes  leads to other issues unrelated to the new format.

Currently other support issues arise when a library that depends on another library is included by a user. The new properties system should allow us to ensure a more intelligent #include system either automatically including other libraries or by prompting the user to do so.

The new system has a lot of future flexibility (potential) such as supporting "auto download" of missing dependencies and better version control.

I am not suggesting the new system is perfect but the clarity it brings to the UI looks GREAT and the reduction in potential support problems is a big plus.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 22, 2013, 07:34 pm
VisualMicro,
I fully understand the goal of a better library & build process.
The problem is that the goals you mention cannot be achieved with the current specification.
Yes it can work for simpler libraries, but not for more complex libraries or libraries
that have different maintainers for the different architectures.
Have a look at the specific examples I provided earlier.
Still not convinced, have a look at the current 1.5.4 examples.
Go down to the USB examples. They show up on an UNO and yet will create compile errors
because the UNO does not have support for USB and yet the users still sees the examples
that depend on USB support.

The .properties library specification does not have the needed capabilities
to tie libraries/examples/sketches to specific cores or more importantly specific chips within a core.
The current specification simply lumps together the capabilities of all the architectures
within a core. This is unrealistic given the vast differences between the chips that are
being supported within a core like the stock AVR "Arduino" core.

So while I admire and agree with the goals of the new .properties specification, it is a long way
from providing the desired/needed capabilities.

So far away, that I think it needs a complete reboot/restart.

As I said earlier, the current spec solves a few issues, but also creates new ones, and
leaves many current real-world 3rd party issues un-resolved.

My view is that if something like this is going to implemented and be embraced,
then it needs to take a bigger step in solving issues for everyone:
- Arduino Team developers
- users
- 3rd party library developers
- 3rd party tool developers

The problem I see with the current definition is that it was done in a vacuum without the input
of the 3rd party library and tool developers.
Because of this, we are already seeing a luke warm response towards it and even examples
where it doesn't work.
Come on Arduino Team, it is time for you guys to embrace being more open so you can actually
take advantage of the 3rd party community resources rather than having to battle them.

My view is that to come up with a system that solves many of these issues
will take the input of many people.
It can't be done by the Arduino team by itself  or just by the 3rd party developers as many of the issues
for each are not seen or experienced by the others.
I also believe that many things will have to be dealt with simultaneously.
In other words, to make something like this really work, it isn't possible to be just about a library spec or a directory
structure. It also requires re-examining the actual build process itself as many of these issues actually
stem from that.


--- bill
Title: Re: Comments on Arduino 1.5 Specifications
Post by: visualmicro on Oct 22, 2013, 08:05 pm
Hi Bill,

Yes a fuller specification would be ideal and way to more easily tune the options for the hardware will remove tons of confusion.

With 1.0.x at end of life and with continued support for pre 1.5 libraries in 1.5, I hope we can just stick with it for a while and take time to discuss a better solution or extended solution.

I hope that final solution takes into account all of the points you make so eloquently and caters better for duplicate library source code names.

Tim

edited for clarity
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 22, 2013, 08:11 pm
Bill

Thank you for summarising and pointing the real need for discussion.

Let's hope the Arduino team listens...
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 22, 2013, 08:32 pm
Bill, while you're made some good points, you've also used some pretty strong language.

In particular, I would disagree with your opinion that this spec was "done in a vacuum without the input of the 3rd party library and tool developers".  Months ago, Cristian did post this spec to the developers mail list.  Input was requested.  How much useful input was actually given, and to what extent that input shaped the final spec is a good question, but it was far from "in a vacuum without the input".

I think your point about the technical limitations could be better made with less harsh rhetoric.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 22, 2013, 08:41 pm
Also, I just use the term "final spec", but honestly I have no idea if the spec really is finalized or open to changes?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 22, 2013, 09:32 pm
Paul,
I concede that perhaps my comments were a bit harsh and bit ill informed.
But over the past few years even you have been quite frustrated with the Arduino teams lack of willingness to
work with and accept ideas (or even code) from outside of its inner circle.

My biggest concern is that this will end up creating a situation like what happened with
final release 1.0
What the team did then with respect to backward compatibility was simply inexcusable
as it wasn't necessary.
My concerns would be greatly relieved if some one from the Arduino would
commit to not repeating that mistake.

--- bill
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 22, 2013, 09:53 pm
I launched this thread many, many months ago, back on: May 11, 2013, 11:53:48 am, as soon as I read the specifications. Jantje and Paul joined the conversation the very same day.

But we had to wait till... September until a member of the Arduino team answered. That is, 4 months later.

Maybe I'm wrong about this forum and maybe there's another secret one.

Now, I'm not convinced at all by the last minute solution approach. Just consider what happened in the U.S. with the government shut-down ;) till next January-February.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 22, 2013, 10:57 pm
Yes, I've had many extremely frustrating occasions try to contribute to Arduino.

Since then, the leadership has changed.  I'm not saying things are perfect now.  But I do consider the worse of those experience to be old and in the past, now that Cristian is the technical lead for Arduino.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 23, 2013, 12:05 am
I know it is not constructive but I can't help it.
http://www.youtube.com/watch?v=VkCDxu6Z2XE (http://www.youtube.com/watch?v=VkCDxu6Z2XE)
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 23, 2013, 12:03 pm
Bill!

First I want to reassure you that if you want to stick to the 1.0 lib format, you can safely do that, we are not going to remove old library support. We already discussed that (in a google issue maybe? I can't remember), so I don't know why you're so concerned about that yet.

You're right about the fact that the library properties doesn't manage specific chips inside an architecture, we deliberately made this decision, otherwise we would end up with a structure like arch/[architecture]/[cpu] (and probably jantje would kill me :)), for example arch/avr/atmega328: this lead to massive code duplication since code for atmega328 is very similar to code for many other atmegaxxx, sometimes only a single line of code or a register name is different, and that can be handled easily with #ifdefs.
The same is not true when you move your focus from CPU to ARCHITECTURES: different architectures like SAM and AVR have very low code matching, practically null, and you end up using #ifdefs to select the whole file: in this case is much more clear and handy to use different files at all.

May we could extend just the .properties file to say that a library runs only on a specific chip? so for example a library for a 32u4 would not be listed when the selected board doesn't have a 32u4? Maybe.
We could extend that also to the examples? Maybe.
I would like to see a proposal from you on how you will do that.
These are all things that can be added (also later) to extend the capabilities of the library specification. BTW, this is all parsing work that will go to the back of tools-maintainers too.

"dependency" and "dependency-core" are placeholders for the future library manager. The version specified in the dependency-core is the  version of the CORE (and not the IDE as you presumed, even if they are still coupled). You can find the core version inside the platform.txt file. As you can see those fields are just drafted and there is room for improvements here when the library manager will be put in place. I'll make an announcement on the developers list when we start to define it.
At the moment the IDE doesn't enforce dependency check on those fields. I'm happy to see how you would improve that, consider that they are mainly focused for machine-dependency check on the future library manager.

About compatibility of 1.5 library with IDE 1.0: its true, but again, how would you change the specification to allow that? See that there is very small room for improvements on the old lib-1.0 format.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 23, 2013, 12:14 pm

I launched this thread many, many months ago, back on: May 11, 2013, 11:53:48 am, as soon as I read the specifications. Jantje and Paul joined the conversation the very same day.

But we had to wait till... September until a member of the Arduino team answered. That is, 4 months later.


I'm sorry to have joined this conversation so late, unfortunately I didn't noticed it until September, otherwise I'll have answered it early.

BTW I've sent the library specifications on the developers list on Feb. Why don't you replied, or started this topic there?
If you really have at hearth this topic, why don't you even tried to discuss it on the developers list, or maybe send an email to me or someone else of the Arduino Team?

For the next time, I'll praise you, to discuss this kind of things on the developers list, to be sure that someone of the Arduino Team, but also the whole developer community, notice it, not only the Arduino Forum's attenders.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: visualmicro on Oct 23, 2013, 12:17 pm
@avenue33 Do you know about the developers site? This is useful for Ide and build process discussion.

https://groups.google.com/a/arduino.cc/forum/#!forum/developers (https://groups.google.com/a/arduino.cc/forum/#!forum/developers)
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 23, 2013, 12:17 pm

Also, I just use the term "final spec", but honestly I have no idea if the spec really is finalized or open to changes?


We are still in beta, so... theoretically everything can happen. I'm thinking to reopen the discussion on the developers list.


I'm not saying things are perfect now.  But I do consider the worse of those experience to be old and in the past, now that Cristian is the technical lead for Arduino.


Thanks Paul. This is really much appreciated.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 23, 2013, 12:45 pm
Thanks for the link.

Actually, I didn't know there was a developer site for Arduino on Google, as the code is hosted at GitHub and the forum is here. Quite confusing!

I'm on Twitter, Facebook, LinkedIn, GitHub, ... but I closed all my Google-related accounts last year.

Now, I've opened a new Google account and joined the developer site, on time for next Arduino 2.x revision.

Nevertheless, it seems that other developers duly introduced to the developer site at Google weren't listened at.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 23, 2013, 01:45 pm

.. (and probably jantje would kill me :)),..

Don't worry. I'm really good at architecture/analyses/design/problem solving but my skills for killing are lacking.
So you are safe due to a lack of skills on my side.  ]:D
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 23, 2013, 06:51 pm
Guys,

Since you're requesting it, let's go talking about the library specification on the developers mailing list.
To let everyone subscribe on the developers list I'll start a new thread about that in a day or two.

Here the link to subscribe:

https://groups.google.com/a/arduino.cc/forum/#!forum/developers

Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 23, 2013, 06:54 pm

Bill!

First I want to reassure you that if you want to stick to the 1.0 lib format, you can safely do that, we are not going to remove old library support. We already discussed that (in a google issue maybe? I can't remember), so I don't know why you're so concerned about that yet.

Because of history and because of various comments like what you just said:
Quote
why you're so concerned about that yet.


Eventually "yet" arrives, and from my perspective, it was poorly handled
on the 1.0 release.
When 1.0 was released,
several things were changed at the last minute that broke
backward compatibility with existing libraries.
100% of the 3rd party library code that was stable for MANY years and was working
on all the beta releases of 1.0 was suddenly made obsolete and no longer compiled
on the final 1.0 release.
The Arduino team deliberately made the decision to break 100% of the existing 3rd party library code
to force everyone to update their libraries. There were easy ways to make most of the existing code
continue to work, but the Arduino team decided against it.
This is an example of a decision that didn't impact the library code
included in the Arduino release but had a dramatic impact on all the 3rd party code
and developers.
We are still fighting issues related to this.
This why I have such concerns.


You're right about the fact that the library properties doesn't manage specific chips inside an architecture, we deliberately made this decision, otherwise we would end up with a structure like arch/[architecture]/[cpu] (and probably jantje would kill me :)), for example arch/avr/atmega328: this lead to massive code duplication since code for atmega328 is very similar to code for many other atmegaxxx, sometimes only a single line of code or a register name is different, and that can be handled easily with #ifdefs.
The same is not true when you move your focus from CPU to ARCHITECTURES: different architectures like SAM and AVR have very low code matching, practically null, and you end up using #ifdefs to select the whole file: in this case is much more clear and handy to use different files at all.


It isn't quite that clean and simple.
What happens when you have multiple vendor cores that support the same CPU Architecture?
The current .properties library spec doesn't seem to handle this.
i.e. think, maniacbug1284p, attiny, Teensy
All those are "AVR" but yet are different vendor/maintainer cores.

I also think that it should not be assumed that code differences are primarily due to low level hardware differences.
There are cases where the differences are due to the toolset related to an architecture
and are a s/w only compatibility matter.
In those cases you can have a similar situation as to what you mentioned above, using
an ifdef around minor code differences.
For example, there can be slight differences as to how AVR vs ARM vs pic32 handle
their standard i/o which can affect a library.

tool/board/architecture/chip portability mapping is never an easy thing to resolve
and people will always have differing opinions about what methodology works better.

My concern is that not enough thought has gone into the new library design to make it really useful
in 3rd party environments.
I'm concerned that the 3rd party guys may end up with a system that is different but still doesn't
solve many of their needs.
To a large extent I think this effect is quite natural in that the Arduino team is focused
on their release - as they should be - so there isn't really a focus
on how to help out the 3rd party developers, which might even be competitors.
I do believe that In some cases there may be situations where a given methodology may
work better for the 3rd parties but have no benefit to the official Arduino code.
If that methodology creates more work for the Arduino team, it may be a
difficult hurdle for the 3rd party community to convince the Arduino
team to pursue that path.

I brought some of this up earlier but here is some additional detail
a few items that seem problematic.
For example, how do other cores really fit into this?
i.e. what about other cores that are in use like maniacbug1284p, or attiny?
Both of those are AVR architectures, but the current .properties spec doesn't appear to have a
way to deal with this.
The 1.5 hardware spec clearly shows how different vendors create their own subdirectory and
then can create architectures below. But in the .properties spec, there is no reference to these
core "vendors"/maintainers
Also the related 1.5 hardware spec doesn't go into details on where the vendor should put his tools
for his architectures.
Currently, the "Arduino" vendor that ships with the IDE,
puts it's build tools in the  "tools" directory parallel to the "vendor/maintainer" directory.
Having integrated the chipKIT cores and tools into an early 1.5 environment,
I found it worked better to put the build tools under the actual architecture directory to keep everything together.
For chipKIT there were also issues that required this given that some of the tools overlapped but
were actually different.
Having the all the build tools under the architecture directory allowed the chipKIT support to be
added into Arduino 1.5 by simply copying over a single directory.
And BTW, when the chipKIT support was integrated, chipKIT boards were using arduino 0023 vs Arduino cores using arduino 1.5
all under the same 1.5 IDE.
This is an example of how cores can get skewed and a reason to consider this.
I believe it is naive to believe or require that all the architecture cores must remain in full sync with the IDE.
There needs to be a decoupling of the IDE from the tools and and the cores.


May we could extend just the .properties file to say that a library runs only on a specific chip? so for example a library for a 32u4 would not be listed when the selected board doesn't have a 32u4? Maybe.

If this approach moves forward, then I think that the .properties should support many
of the properties from the board.txt file.
That way a library or sketch (assuming there is an optional .properties for sketches) could specify anything
it wants as a dependency


We could extend that also to the examples? Maybe.
I would like to see a proposal from you on how you will do that.
These are all things that can be added (also later) to extend the capabilities of the library specification. BTW, this is all parsing work that will go to the back of tools-maintainers too.

It depends. For example, if much of this property stuff is just to create pretty/clean lists of
libraries or examples in the IDE GUI for the user, then no, the 3rd party tools may not have to support this since
they may have different way of specifying libraries.
It would be nice to have a system that could benefit the Arduino IDE but doesn't unduly burden the 3rd party
tools that didn't need the capability.


"dependency" and "dependency-core" are placeholders for the future library manager. The version specified in the dependency-core is the  version of the CORE (and not the IDE as you presumed, even if they are still coupled). You can find the core version inside the platform.txt file. As you can see those fields are just drafted and there is room for improvements here when the library manager will be put in place. I'll make an announcement on the developers list when we start to define it.
At the moment the IDE doesn't enforce dependency check on those fields. I'm happy to see how you would improve that, consider that they are mainly focused for machine-dependency check on the future library manager.


But what does "CORE" really mean?
What happens when you have other 3rd party cores?
It doesn't appear that it is possible to specify that a  library only works on that core?
i.e. suppose the library wants to specify that it only works on Teensy or maniacbug1284p cores?


Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 23, 2013, 06:55 pm
Quote
About compatibility of 1.5 library with IDE 1.0: its true, but again, how would you change the specification to allow that? See that there is very small room for improvements on the old lib-1.0 format.


I think that what could go a long way is to ensure that very simple 1.5 libraries
still work on 1.0 because I don't see everyone immediately jumping on the 1.5 bandwagon.
By creating this capability, you could get a quicker adoption of the 1.5 library format by
library developers since they could now create and maintain a library that works on both
1.x and 1.5 without having to avoid the newer 1.5 format.

For example, if a 1.0 library could simply add a .properties file and
the common code didn't have to be in src then simple 1.5 libraries
could still work on 1.0 but 1.5 could get and use the additional information
from the .properties file.

OR as an alternative (and I mentioned this earlier as well),
the IDE/library manager should maintain its ".properties" stuff separately from what
may be provided by the library. So the IDE takes some information
from the library and uses it to install the library in the IDE.
Or another option for importing/installing the pre 1.5 library files would be the query the user
for the information and create the needed IDE ".properties"  stuff during
the install process.

I think many things are possible and I'll be happy to participate in the process
of coming up with solutions.
I also think that there are several others that have responded in this thread
that have an interest in participating to come up with something better.

The key is coming up with a list of wants & needs for this type of capability.
And people need to be very clear at understanding the difference between
their true needs and their wants.
Once there is discussion and a exchange of these ideas, it becomes much
easier to decide on the how to proceed: Try to cover all the needs first,
then try to integrate as many of the wants as you can given the
timing and resources available.

--- bill

Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 23, 2013, 06:55 pm
Cool,
Just noticed the post about the developers thread.
We can all continue over there.

EDIT:
I do have a bit of an off topic question:
Why use Google Groups? Why not have the discussion over here on the Arduino Forum?

--- bill
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 23, 2013, 07:33 pm

Guys,

Since you're requesting it, let's go talking about the library specification on the developers mailing list.
To let everyone subscribe on the developers list I'll start a new thread about that in a day or two.

Here the link to subscribe:

https://groups.google.com/a/arduino.cc/forum/#!forum/developers


Ok, initial post awaiting approval.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 23, 2013, 07:45 pm
Maybe the goals & purpose of this library spec aren't clear?

Bill's been talking a lot about whether the IDE offers incompatible libraries and examples.  But I can't find any mention of that actually being an intended goal.  If the metadata is going to be used to filter what libraries, examples and other options are offered to users, I can't find anything written about how that's supposed to work.

The spec does talk about 2 goals: #1: facilitating a library manager that automatically downloads libraries from a central server, and #2: supporting multiple architectures.


Regarding multi-arch support, I must agree with Bill, requiring a directory structure that isn't compatible with 1.0.X is obviously going to lead to very slow adoption of this spec.  As the author of several libraries and the current maintainer of several more, it's pretty easy to choose between publishing in a format that works on all widely used Arduino IDEs, versus a format that only works on the newest.

Cristian, I'm really glad you said this:

Quote

First I want to reassure you that if you want to stick to the 1.0 lib format, you can safely do that, we are not going to remove old library support. We already discussed that (in a google issue maybe? I can't remember), so I don't know why you're so concerned about that yet.


I'd like to point out some language near the end of the spec, which might be the source of concern:

Quote

At some point in the future, the IDE will drop pre-1.5 library support. There is no specific timeline for this, it will be determined once the new library format has become widespread.


Maybe this part of the spec should be updated, if you're really committed to keeping the old format supported?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 23, 2013, 07:55 pm
Whether it is off-topic or not (I think it is on topic). but:

I do have a bit of an off topic question:
Why use Google Groups? Why not have the discussion over here on the Arduino Forum?



Actually, I didn't know there was a developer site for Arduino on Google, as the code is hosted at GitHub and the forum is here. Quite confusing!


I knew about the developers list but I was not aware you could reread the list on google. To me mailing lists are terrible for discussions.
It seems the new web site does mention the google groups where the old one only mentioned the mailing list.
So you are not the only one asking the question.

To be honest: As the Arduino core team has not shared any documented process on releasing specs; I feel any remark like "you should have commented on the developers mailing list." is not constructive. Where does it say "that is the way?" And why is there a forum named "Development :: Suggestions for the Arduino Project" if suggestions for improvement are not to be posted here?

I know I'm quite a formal guy and the open source is mostly anti-formal. But as you can see: people ask "How could I have known?" the only correct answer is "Luck" and "you can increase you Luck by increasing your attendance".
Just to state the obvious: This has nothing to do with cmaglie but with the typical growing pains of any organisation.

On topic: I made a script that adds the ifdefs. That makes it work. But the script is unix only for now and it is a shame it doesn't work out of the box. Maybe Arduino can consider running this script before making the installation files. That has the benefit that the ifdefs are not there for the "library developers".
To be honest: I don't expect a positive response because the packaging process is mostly already complicated and it may also impact testing.

Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on Oct 23, 2013, 08:05 pm
Jantje

Thank you for saying loud what I'm thinking actually. This is just another example of what I've already pointed at, the lack of consideration and lack of respect towards Arduino contributors from the Arduino team .

Anyway, the question is now: How to use standard tools as GCC to compile a sketch with the new library structure?

The Arduino IDE avoids the issue by performing under-the-hood work. I could have tried to mimic that on Xcode but, let's face it: Wouldn't be easier to consider a sketch as a plain C++ project?

When Arduino is going to understand that, behind the limited IDE, there's a framework worth considering and promoting?

Now, there are so many interesting boards outside, cheaper and more powerful, ...  
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 23, 2013, 08:49 pm

Maybe the goals & purpose of this library spec aren't clear?

Bill's been talking a lot about whether the IDE offers incompatible libraries and examples.  But I can't find any mention of that actually being an intended goal.  If the metadata is going to be used to filter what libraries, examples and other options are offered to users, I can't find anything written about how that's supposed to work.

The spec does talk about 2 goals: #1: facilitating a library manager that automatically downloads libraries from a central server, and #2: supporting multiple architectures.

Paul,
Directly from the 1.5 library spec page in the IDE behavior section:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification (https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification)
Quote
When the user chooses a board the IDE automatically selects the correct library implementation for the board's architecture. If a library doesn't provide an implementation for the currently selected architecture, the library is not displayed cannot be selected. Examples are shown only if the library has an implementation for the currently selected board architecture.

And in Cristian's first post to this thread:
http://forum.arduino.cc//index.php?topic=165827.msg1384527#msg1384527 (http://forum.arduino.cc//index.php?topic=165827.msg1384527#msg1384527)
Quote
Also note the with the library.properties you can filter the library from being displayed/compiled on architectures that are not supported by the library itself.


Form that language I interpret it to be a desired intent/goal to have the IDE filter out
libraries and sketches that are not supported on the selected board.
In fact you see this behavior already today in the 1.5.4. IDE.
Select a DUE board, then click
[File]->[Examples] ->[Scheduler]->MotorKnob
This is an example in the Scheduler library.
Now change the board to "Arduino Uno".
The Scheduler library is now gone.
The reverse happens for the "Bridge" library which is AVR only.
I can go in and modify the architectures value in the .properties
file for the library and the IDE uses the information.
So clearly the intent is that the IDE is going to use this information
to determine what is filtered for the user.

While I think it is a good goal and would be a great capability for the IDE
to support for its Arduino users, the information in the .properties file outlined in
the current 1.5 library spec is simply not capable of providing that capability
across the 3rd party cores.
And that is why I brought up all the real-world examples of how the
current spec falls down.


Quote

Regarding multi-arch support, I must agree with Bill, requiring a directory structure that isn't compatible with 1.0.X is obviously going to lead to very slow adoption of this spec.  As the author of several libraries and the current maintainer of several more, it's pretty easy to choose between publishing in a format that works on all widely used Arduino IDEs, versus a format that only works on the newest.


My point exactly. And that is also part of the reason for my concern.
I worry that the Arduino Team might resort to a heavy handed approach again
to force its adoption like it did in the 1.0 release.

Quote

Cristian, I'm really glad you said this:

Quote

First I want to reassure you that if you want to stick to the 1.0 lib format, you can safely do that, we are not going to remove old library support. We already discussed that (in a google issue maybe? I can't remember), so I don't know why you're so concerned about that yet.


I'd like to point out some language near the end of the spec, which might be the source of concern:

Quote

At some point in the future, the IDE will drop pre-1.5 library support. There is no specific timeline for this, it will be determined once the new library format has become widespread.


Maybe this part of the spec should be updated, if you're really committed to keeping the old format supported?


I totally agree with Paul here. I saw that language very early on when looking over this 1.5 library stuff
many months ago. That language pretty clear states that there is a goal
to remove support for the pre 1.5 libraries.
The first thing I thought is UH OH... Here we go again.
It's going to pre 1.0 vs 1.0 libraries all over again.


--- bill
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 23, 2013, 10:01 pm
Ah, yes, you're right.  I overlooked the "IDE behaviour" section.  It is right there!

As a maker of a 3rd party boards on a different "architecture", this worries me somewhat.  My hope is most libraries will use "architectures=*".  But a very likely scenario is "architectures=avr", even when the AVR specific stuff can be emulated (as Teensy 3.0 does for many commonly used AVR-isms).  I definitely don't want stuff that "just works" made invisible, only because the original library author only tested on a single platform.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 23, 2013, 11:32 pm
Paul,
What is your understanding of the "architectures=" field?
OR what does  architectures=avr really mean?
I brought this up earlier but I think it got lost in the noise.

Suppose the Teensy or maniacbug1284p core is updated to be recognized by Arduino 1.5.4
What then?
I can't resolve in my head what the "architecture" field means given the current 1.5 hardware spec
vs the 1.5 library spec.
The hardware spec talks about vendor/maintainer but the 1.5 library spec
talks about "core-dependencies".
The hardware spec talks about "architectures" which live under a vendor/maintainer directory.
So if the "core-dependencies" in the .properties file refers to a vendor/maintainer core and
its sub architectures, how does a library refer to a specific architecture within a specific vendor/maintainer core?
i.e. how would a library specify that it only runs on the Teensy or maniacbug1284p core?
OR what about a core that works on any AVR architecture processor core regardless
of the vendor/maintainer:
Arduino, Teensy, manicbug1284p, Attiny.

In other words how do 3rd party cores get installed in 1.5 and
then how do libraries specify their dependency on these 3rd party cores or their
sub architectures using the .properties file?

And for products like Teensy that include Teensyduino.
Within Teensyduino,  is "teensy"  the vendor/Maintainer with sub architectures of avr and arm?
And if so, how does a library .properties file for an "AVR only" library
specify that it can only run on the boards that use the Teensy avr core?

As it is today, it does not appear that the .properties file can handle
3rd party cores dependencies very well.

One possible solution might be to alter the architectures field.
Perhaps something like:
architectures: vendor/architecture (optional-version)

That could potentially even allow something like:
architectures: */avr
To specify any vendor that has an "avr" architecture.

--- bill

Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 23, 2013, 11:35 pm
When I started to look at the implementation of the library spec in 1.5.4 (I mean look at the sub folders of libraries) I was struck by the tag
Quote
"architectures=*".

It goes with what Paul already stated several times about "There is not 1 person maintaining the library".
Basically the properties file is common. So it depends on 1 person to maintain. That one person can decide on which platforms a library can/may run  :smiley-eek:.
The same counts for other properties.
What struck me the most was that Arduino mostly works on folder names (like for avr and sam in the hardware/arduino folder) and here decides to go for a "described in file" specification. And that file is under control of Arduino  :smiley-eek:.
The reasoning is at least "strange"  $).

And then I come to something difficult to explain. I mean there are facts (like It is unnecessary hard to implement in eclipse ) and there are ideas behind a approach (Like it should be easy for the library builders and we don't care about tool builders).
The document explains (though not explicit enough to me) the aim but not the reasoning/ideas behind the "solution(s)".

In this particular case my "idea behind the solution" is very oo oriented.
First of all :There are many things that don't fit the model  I'll describe; so I'm sure this model was not in mind when the design was done.
In a oo world (surely C++) there is definition and implementation. so I see a definition which is in the src folder  :smiley-eek: and a architectural implementation source code is in the lib/[arch] folder  :smiley-eek:. (the  :smiley-eek: for the folder names)
That would mean that the implementation references the definition but the spec says
Quote
For example, if we try to compile the above library for a board based on AVR architecture, the IDE will automatically compile and include the following folders:

Which I understand as both folders are  part of the build path and the include path. Why would I need 2 folders in build and include when 1 is implementation and one is definition?
In this same idea there should be a properties file for each implementation stating which architecture it supports. If there is a conflict the specific setting should overrule any more generic setting as we do with boards.txt and platform.txt.
That solves the "I write a definition but I don't know the implementation" problem but introduces a new one.
What if 2 (or more) library implementations state they can implement your boards architecture?
The most common sense would be that the end user can select the "wanted" library implementation.
This can very easily be implemented by having a define in the properties file. In that case selecting a library implementation means adding a define to the compile path and including the correct header.
This also solves (partly) the forced code duplication problem the current spec has. This because a single folder can hold endless architectures.
I know Arduino has something with ifdef but I feel this is a more robust and easy to implement solution covering more cases as the current spec.
I also realize this is completely incompatible with the current folder names. For instance the current [architecture] folder is prbably better a [implementer] folder. The src folder can not contain code but only definition....

As we are at it.
One of the things I liked about the Arduino libraries (pre 1.5) is that 1 library=1 include in the ino file.
The 1.5.4 includes all the headers in the src folder.  for 1 library (I don't remember its name) there is even no file with the library name. I think this will confuse the newbe (and puzzle  even the more experienced like me).
My preference goes to 1 include in the ino file with the library name (The name of the folder and maybe preceding folders) for 1 library.  

Jantje


Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 23, 2013, 11:57 pm
bill
I guess you are asking the wrong person on how it is currently implemented.
The only more or less 1.5 compatible teensy right now has been build by me.  (off course with help of Paul)
As far as I remember (I don't use the ide) it works in the IDE for Teensy2 and Teensy2++ but surely not for Teensy3.
It is still setup in the pre 1.5.3 format (that is without the library specification we are discussing here) so all libraries are under hardware/teensy. Also all executable code is there. So I used teensy as vendor. (pretty obvious)

Paul preferred (like I do) to only have 1 "hardware folder" so I named it all. So in my implementation teensy hardware is called all.
In the oo spirit (I also elaborated about in my previous post)  my plugin allows boards.txt to overwrite all!!!!!!! settings in platform.txt.
This allows me to overwrite even the compiler command. Which is needed for Teensy3.
However Teensy3 does not make a library so I had to add a special flag which is not supported by the arduino IDE.



On the hardware/[arch] folder
I never really understood the need for an [architecture] folder. I mean
When a artist buys a due because he has been told he needs that board. Does he care it is sam or avr when he doesn't even know what that means? (I don't know what sam and avr means)
I think it is not compliant with following points in the library guide line
Quote
Be kind to the end user.
Match your API to the underlying capabilities. You don't want to expose implementation details to the user
Organize your public functions around the data and functionality that the user wants.
Use full, everyday words.

But it is not a library? or is it? ]:D

Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 24, 2013, 12:57 am

bill
I guess you are asking the wrong person on how it is currently implemented.
What I was trying to solicit is to  see is if everybody else is seeing the flaws
in the current .properties specification with respect to trying to specify 3rd party cores
or architectures within those 3rd party cores.


The only more or less 1.5 compatible teensy right now has been build by me.  (off course with help of Paul)
As far as I remember (I don't use the ide) it works in the IDE for Teensy2 and Teensy2++ but surely not for Teensy3.
It is still setup in the pre 1.5.3 format (that is without the library specification we are discussing here) so all libraries are under hardware/teensy. Also all executable code is there. So I used teensy as vendor. (pretty obvious)


I did similar for chipKIT a year ago.
I have a working chipKIT environment that works with the Arduino 1.5.1 IDE GUI.
It wasn't that big of deal since the libraries were all separate at that time.
This allowed chipKIT to be using 0023 while the avr and sam used 1.5.1




Paul preferred (like I do) to only have 1 "hardware folder" so I named it all. So in my implementation teensy hardware is called all.
In the oo spirit (I also elaborated about in my previous post)  my plugin allows boards.txt to overwrite all!!!!!!! settings in platform.txt.
This allows me to overwrite even the compiler command. Which is needed for Teensy3.
However Teensy3 does not make a library so I had to add a special flag which is not supported by the arduino IDE.

I didn't follow that.
You can do pretty much anything to the compiler, linker, and download commands given the
flexibility offered by the platform and board files. I had to completely change things for chipKIT but the flexibility was
there for me to change all the commands around when chipKIT boards were selected without
having to change anything in the Arduino JAVA IDE code.



On the hardware/[arch] folder
I never really understood the need for an [architecture] folder. I mean
When a artist buys a due because he has been told he needs that board. Does he care it is sam or avr when he doesn't even know what that means? (I don't know what sam and avr means)

While agree that end users should have not to know too much low level details,
I didn't follow the rest of your comment about "architectures"

At a minimum, there as to be separate areas for the different toolsets given multiple toolsets
are required to develop across the different processors.
As far as the end user goes, most of what are discussing is completely hidden from
the normal end user.
The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.
But to do that the library writer and core developer has be very familiar with these
differences and how to configure and specify what works together
AND the system must be capable of allowing enough information to be specified
to ensure that the system can select the proper tools and libraries once the user
specifies which board he is using.

In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.

What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.

I also believe that it is more important to resolve the functional issues
before the implementation issues.




Title: Re: Comments on Arduino 1.5 Specifications
Post by: Jantje on Oct 24, 2013, 01:30 am
Quote
What I was trying to solicit is to  see is if everybody else is seeing the flaws
in the current .properties specification with respect to trying to specify 3rd party cores
or architectures within those 3rd party cores.

I think we all see some flaws. Not sure we all see the same.
I was also wondering whether anyone represented the ino tool.

Quote
I didn't follow that.
You can do pretty much anything to the compiler, linker, and download commands given the
flexibility offered by the platform and board files. I had to completely change things for chipKIT but the flexibility was
there for me to change all the commands around when chipKIT boards were selected without
having to change anything in the Arduino JAVA IDE code.

What I mean is that if platform.txt states
compiler.c.cmd=avr-gcc
and you selected a UNO and the boards.txt file contains
uno.compiler.c.cmd=sam-gcc
the plugin will use sam-gcc while I expect the Arduino IDE to use avr-gcc

I fully agree the boards.txt and platform.txt makes the life of a tools developer for the compilation part easier.
There has already been a discussion on the developers list that the same should be implemented for the upload.

Quote
The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.

It is visible in 1.5.2 when you select a board

Quote
In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.

I fully agree on this but there is to much focus on Architecture while this could solve many other problems as well.
For instance this http://forum.arduino.cc//index.php?topic=143940.msg1081073#msg1081073 (http://forum.arduino.cc//index.php?topic=143940.msg1081073#msg1081073)
There is a library for avr for tone and you have a new implementation which is better but takes more space and is still in beta.
Some people will want the old one. other will want the new one. Why not have both and let the user select the most appropriate for him/her?
In this particular case the folder name can not be the architecture as the architecture is the same for the 2.
so you will have tone/arch/stable and tone/arch/beta
the folder /arch/stable contains a property file stating the architectures it runs on and so does arch/beta.
So it can be that /arch/stable supports avr, sam, all and chipKIT but arch/beta only avr.

Quote
What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.

Technically yes but it misses on process and ownership.
For example if I created a new tone library I want to add a folder under libraries/tone/arch/ and it should work.
I don't want to be dependent on someone else to change the existing properties file (for instance because this library supports sam implementation which is not yet available). And when the folder is removed the sam implementation is removed as well.
In other words: As a provider of the library I want to be able to specify some fields in properties. The best way to do so is add a properties file which overrules (or adds to) the properties in the parent.
I don't want to be dependent on a tool (which I would need to implement in eclipse) that updates the properties file based on user import and export activity.

Best regards
Jantje
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 24, 2013, 03:19 am

Paul,
What is your understanding of the "architectures=" field?


I honestly do not really know at this point.

Ultimately, my top priority is making everything "just work" on Teensy.  A strong secondary goal is enhancing the Arduino API, ideally in ways compatible with all official Arduino boards, like the advanced ADC API recently discussed on the developer mail list.

I also do want to utilize this multi-platform stuff, but that's a secondary goal.  If there's any sort of trade-off, making everything "just work" for customers takes precedence.  I already apply lots of patches to the IDE.  I want to do things in the simplest, most maintainable ways, but if patching results in more work for me but a better experience for my customers, I'll always optimize for end user experience.

Regardless of what the spec says, I believe what this really ends up meaning will largely depend on how many library authors actually end up using it.  I've never been good at predicting the future, so I'm going to resist the urge to make any more predictions.  I'm mostly in "wait and see" mode.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 24, 2013, 03:55 am

What I mean is that if platform.txt states
compiler.c.cmd=avr-gcc
and you selected a UNO and the boards.txt file contains
uno.compiler.c.cmd=sam-gcc
the plugin will use sam-gcc while I expect the Arduino IDE to use avr-gcc

I guess I don't have that expectation and don't see an issue here.
The  point of having the user pick a board is to potentially do things that are board (chip) specific,
to allow a means to potentially do something differently for a given  board (chip).
If there are different chips supported within the same chip architecture,
you must be able to potentially override any default settings in the architecture (platform.txt) file.
While the above is a very contorted example, the values of the fields in the board file for
a particular board have to be able to override the field values in the platform file.
In my view, what you have described is the way it should and needs to work.

For example, maybe  I want to specify a shell wrapper to set up my tools
for source level debugging. So maybe I set up a board type that uses a shell
wrapper script that sets up link maps, DDD, AVaRICE, and my AVR dragon as part
of the build and/or upload process. I might have to override the platform.txt values to
get it to call my wrappers instead of the calling the tools directly.
(I've done this, and  did this to get chipKIT up and working on IDE 1.5.1)


Quote
The user doesn't have to know any of this as long the system allows a library
to specify which environments it works in.

It is visible in 1.5.2 when you select a board


I think perhaps you have misunderstood me.
When there is a mechanism in place that allows a library to specify which boards (chips)
it works on, the user only has to know what board he has. How it works under
the covers is no concern to him.


Quote
In the bigger picture, 3rd party Core, Library, and tool writers are not normal users.
They are the ones that create the s/w to make it easy for the normal users.

I fully agree on this but there is to much focus on Architecture while this could solve many other problems as well.
For instance this http://forum.arduino.cc//index.php?topic=143940.msg1081073#msg1081073 (http://forum.arduino.cc//index.php?topic=143940.msg1081073#msg1081073)
There is a library for avr for tone and you have a new implementation which is better but takes more space and is still in beta.
Some people will want the old one. other will want the new one. Why not have both and let the user select the most appropriate for him/her?
In this particular case the folder name can not be the architecture as the architecture is the same for the 2.
so you will have tone/arch/stable and tone/arch/beta
the folder /arch/stable contains a property file stating the architectures it runs on and so does arch/beta.
So it can be that /arch/stable supports avr, sam, all and chipKIT but arch/beta only avr.

I think the specific library (NewTone) is a poor example, as it used a different name and different API
that the library it is trying to replace and as such does not conflict with the IDE supplied tone library.
I'll assume you mean that you want to have a new/optional/experimental library that uses the same name and API
as some other library.
That said, I'm not understanding if you are wanting the ability to select which library to use
at the sketch level (sketch A used old and sketch B uses new) or whether you simply
want the ability to let the user configure/install a library than can override an existing library
for a given vendor core & architecture.
I brought up a similar scenario in my very first post of this thread.
It was scenario #2 of this response:
http://forum.arduino.cc//index.php?topic=165827.msg1437604#msg1437604 (http://forum.arduino.cc//index.php?topic=165827.msg1437604#msg1437604)
Given that the user libraries already override the IDE supplied  libraries,
the ability to support this capability should be possible if proper dependencies exist in the .properties files.
i.e. Already today, if you install a library in your user sketchbook/libraries area it will override
a library provided by the IDE when the names are the same.
So if a future build mechanism/system maintains this capability as well as properly picking the appropriate
library to support the chip on the selected board, by looking at the various properties
in the library hierarchy, then it all will "just work".
The key is to properly define the .properties file and how the library hierarchy works.
The hierarchy mechanism should use a directory structure that doesn't require 3rd parties
to have to merge their separate properties information.


Quote
What I'm trying to bring up is that from a purely functional point of view
(ignore how ugly or painful it might be to actually implement), the current
.properties spec does provide enough information to specify which
cores and sub architectures within a core are supported by the library when
3rd party cores are involved or when a library is specific to a particular chip.
And without those types of capabilities,
it is impossible to provide a nice clean & simple user experience.

Technically yes but it misses on process and ownership.
For example if I created a new tone library I want to add a folder under libraries/tone/arch/ and it should work.
I don't want to be dependent on someone else to change the existing properties file (for instance because this library supports sam implementation which is not yet available). And when the folder is removed the sam implementation is removed as well.
In other words: As a provider of the library I want to be able to specify some fields in properties. The best way to do so is add a properties file which overrules (or adds to) the properties in the parent.
I don't want to be dependent on a tool (which I would need to implement in eclipse) that updates the properties file based on user import and export activity.
Best regards
Jantje



I totally and completely agree on the need to separate out the property fields  and not require
library developers to have to coordinate and merge their properties and directory trees.
As I mentioned earlier, I believe that requiring 3rd parties to have to cooperate
so closely is unlikely to work since it doesn't scale well across different developers.
To me the current 3rd party cooperation/merging requirement
is a one of the biggest short comings of the current 1.5 library specification.

However, it isn't that simple when it comes to using processor "arch" because
what does "arch" really mean when using 3rd party cores?
Again, think 3rd party cores, not just the cores that come with
the Arduino IDE. Toss in Teensy, maniacBug1284p, and attiny cores.
This HUGE ambiguity over architecture is another point I've brought up several times.
According to the 1.5 hardware spec, which is what supposedly defines how new cores are added,
"arch" is a component of the vendor/maintainer core.
Because of this, there isn't necessarily a single core for the "avr" architecture.
So you can't get away with using just architectures names in the .properties file.
For example, while the Arduino teams supplies the Arduino/avr core,  it doesn't support all the avr chips.
Teensy will be supplying  a partial avr core, and so will maniacbug, and attiny.
A library may only work on the AVR processors that use the Teensy core.
There is no way to specify this.

So again, I believe that it comes back to properly defining the desired functionality first.
In order to do the things like you mentioned above there must first be a requirement to do it,
then there must  be way to actually specify it.
Then,  there can be discussion over implementation,
which includes the directory structure and the mechanisms to support it.
Some of these are inter-related and intertwined and will need to be considered
at that same time.
However, I think trying to jump directly to the directory tree implementation
before agreeing on the functionality much less the way to specify it, is jumping ahead
too far.

--- bill

Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 24, 2013, 10:43 am
I see a certain of confusion about what "architectures" means in the library specification, so let me clarify a bit.

Let's pick the example by bill about maniacBug1284p:

The vendor of this board should proceed this way:

1) create a 3rd party core for his board:

Code: [Select]

hardware/maniacbug/avr/platform.txt
hardware/maniacbug/avr/boards.txt
hardware/maniacbug/avr/cores/arduino/[source code for "arduino" core of maniacbug]


2) create a library that has "platforms=avr" property set.

And he's done.

How the IDE can understand that the library author wants to use the maniacbug core found here:

Code: [Select]

hardware/maniacbug/avr/cores/arduino/[source code for "arduino" core of maniacbug]


instead of the Arduino core found here?

Code: [Select]

hardware/arduino/avr/cores/arduino/[source code for "arduino" core of ARDUINO]


simple: because the user has previuosly select the board "Maniac Bug" (or whatever) from the "Tools" menu.
The definitions for this board are inside the boards.txt under maniacbug hardware folder so the IDE knows that it should use the "arduino" core inside that folder.

The concept of "vendor" is there for this reason, we can have multiple vendors that implements the same architecture!

Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 24, 2013, 10:44 am
Cristian, if you're still reading, here are some questions:

All 22 libraries that ship in Arduino 1.5.4's libraries folder have "core-dependencies=arduino (>=1.5.0)".  What is my Teensyduino installer supposed to do with the core-dependencies field?  If "arduino (>=1.5.0)" means it will only work with a core named "arduino", do I need to rename my core from "teensy" to "arduino"?  Or do I modify the library.properties files to add the name of my core to the properties files?

Likewise, how am I supposed to install Teensy 3.0's non-avr architecture EEPROM library?  Suppose I name Teensy 3.0's architecture "kinetis".  It seems logical that I would add a "kinetis" directory inside libraries/EEPROM/arch and copy my architecture EEPROM.cpp there?  But what do I do about EEPROM.h?  It's currently in libraries/EEPROM/src, shared by all architectures.  My EEPROM library customizes that header, adding a constructor which I use to initialize the hardware (unlike AVR, the eeprom hardware needs to be initialized).  It looks like I'll need to overwrite the platform neutral header with my customized version.  Then, of course there's the library.properties file in EEPROM, which currently says the EEPROM library is only compatible with AVR architecture.  Will I need to modify EEPROM's library.properties, for both architectures and core-dependencies?

SoftwareSerial is another library I've ported to Teensy 3.0.  Fortunately, SoftwareSerial seems to have a header in libraries/SoftwareSerial/src that includes the architecture specific header, but it looks like I might need to patch library.properties, since both architectures and core-dependencies say it's incompatible with my hardware (even if I copy the code into a new directory to make it compatible).  Another complexity with SoftwareSerial is I'll want to patch the examples, which mention the supported pin numbers only for Arduino boards.  I'll want to add similar comments for my 3rd boards.  How am I supposed to do that?

I recall earlier you mentioned the intended path is to contact the library maintainer and get them to update their library.properties file and add the code.  Is that really what you intend for the 22 libraries that ship with Arduino?  Generally Arduino has maintained a policy of facilitating 3rd party hardware, but not shipping any code or file that directly support any non-Arduino boards.  Arduino's lengthy release cycle could also be, well, frustrating if you're the path to get 3rd party support into those library.properties files.

I did notice 1.5.4 has two old style copies of the Wire library, in hardware/arduino/avr/libraries/Wire and hardware/arduino/sam/libraries/Wire.  Maybe this the path?  If I have a library like EEPROM ported to another architecture, but it's already present with library.properties incompatible with my hardware, can I put a pre-1.5 format library in my platform/arch/libraries directory?  Will it override the same-named copy in the main libraries directory?  I was under the impression those directories were the pre-1.5.3 way which was intended to be replaced by library.properties.  Is that right?  If those are the official way for 3rd party boards to be compatible with the 22 "standard" libraries that ship with Arduino, it really ough to at least be mentioned in the library spec.

When I start supporting 1.5.X, I really do want to do it properly.  I honestly do not see what the proper way is to resolve these many issues, at least not without overwriting many library files.  Please let me know your thoughts?
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 24, 2013, 01:27 pm
Paul thanks for trying it, real examples helps a lot my understanding of the issues.


All 22 libraries that ship in Arduino 1.5.4's libraries folder have "core-dependencies=arduino (>=1.5.0)".  What is my Teensyduino installer supposed to do with the core-dependencies field?  If "arduino (>=1.5.0)" means it will only work with a core named "arduino", do I need to rename my core from "teensy" to "arduino"?  Or do I modify the library.properties files to add the name of my core to the properties files?


Well, actually those "*-dependency" fields are not used, and needs to be better defined, so as a temporary workardound you can leave it as is.
Theoretically, "core-dependencies" refers to the core inside your "vendor" folder.
Just to be clear, I imagine (and hope :)) you're doing something like:

Code: [Select]

hardware/pjrc/kinetis/platform.txt
hardware/pjrc/kinetis/boards.txt
hardware/pjrc/kinetis/cores/arduino/....
hardware/pjrc/kinetis/variants/.....
hardware/pjrc/avr/....same stuff here...


the "core-dependencies: arduino" means that you want to use the "arduino" folder inside the "cores" folder of whatever architecture you are using. So unless you named differently the following folder:

Code: [Select]

hardware/pjrc/kinetis/cores/arduino/....


you can leave "arduino" in the core-dependency file.

Quote

Likewise, how am I supposed to install Teensy 3.0's non-avr architecture EEPROM library?  Suppose I name Teensy 3.0's architecture "kinetis".  It seems logical that I would add a "kinetis" directory inside libraries/EEPROM/arch and copy my architecture EEPROM.cpp there?  But what do I do about EEPROM.h?  It's currently in libraries/EEPROM/src, shared by all architectures.  My EEPROM library customizes that header, adding a constructor which I use to initialize the hardware (unlike AVR, the eeprom hardware needs to be initialized).  It looks like I'll need to overwrite the platform neutral header with my customized version.  Then, of course there's the library.properties file in EEPROM, which currently says the EEPROM library is only compatible with AVR architecture.  Will I need to modify EEPROM's library.properties, for both architectures and core-dependencies?


We should update the EEPROM.h and add a constructor. It's a simple fix this time, if you send a pull request I can quickly merge it.
(it would be even better to add a begin() method to avoid the C++ constructors fiasco but... anyway...).
You should append ",kinetis" to the platforms=.... field. It's still not the intended way to work (I mean patching a library with an installer) but I see your concerns about IDE release cycles.
I wanted the platform= property but this example let me think about that. I recall that you already pointed this out, maybe on my first request for comments? Do you a have a solution for that?

Quote

SoftwareSerial is another library I've ported to Teensy 3.0.  Fortunately, SoftwareSerial seems to have a header in libraries/SoftwareSerial/src that includes the architecture specific header, but it looks like I might need to patch library.properties, since both architectures and core-dependencies say it's incompatible with my hardware (even if I copy the code into a new directory to make it compatible).


Same for EEPROM.

Quote

Another complexity with SoftwareSerial is I'll want to patch the examples, which mention the supported pin numbers only for Arduino boards.  I'll want to add similar comments for my 3rd boards.  How am I supposed to do that?


Ok, as said earlier, this is an uncovered part of the specification. Suggestions welcome.

Quote

I recall earlier you mentioned the intended path is to contact the library maintainer and get them to update their library.properties file and add the code.  Is that really what you intend for the 22 libraries that ship with Arduino?  Generally Arduino has maintained a policy of facilitating 3rd party hardware, but not shipping any code or file that directly support any non-Arduino boards.  Arduino's lengthy release cycle could also be, well, frustrating if you're the path to get 3rd party support into those library.properties files.


We need to split libraries from the Arduino IDE and put them in their own repositories, its already in the Arduino plans, but these steps, even if it seems easy, requires time to be done properly. The final goal is to push authors to collaborate, and to make, say, a good SoftwareSerial library that is well maintained with a consistent API across platforms.

Of course, this is my thought, in my ideal world. Instead I see, also from comments in this thread, that current trend is that every author starts its own fork of every library, eventually changing the API, or not following the updates that the upstream library adopt in the meantime.

I know also that is frustrating to wait months for a patch to go in. And I perfectly understand your concerns.

Quote

I did notice 1.5.4 has two old style copies of the Wire library, in hardware/arduino/avr/libraries/Wire and hardware/arduino/sam/libraries/Wire.  Maybe this the path?  If I have a library like EEPROM ported to another architecture, but it's already present with library.properties incompatible with my hardware, can I put a pre-1.5 format library in my platform/arch/libraries directory?  Will it override the same-named copy in the main libraries directory?  I was under the impression those directories were the pre-1.5.3 way which was intended to be replaced by library.properties.  Is that right?  If those are the official way for 3rd party boards to be compatible with the 22 "standard" libraries that ship with Arduino, it really ough to at least be mentioned in the library spec.


I see that the library search priority is another difficult topic.
The current IDE should work, more or less, like this:

Let's call ARCH the architecture of currently selected board, and VENDOR its vendor name.
1) The IDE discards all 1.5 libraries that haven't ARCH in the platform= property.
2) From the remaining libraries it choose the first matching library by searching these folders, in order:
   a) SKETCHBOOK/libraries
   b) ARDUINOPATH/hardware/VENDOR/ARCH/libraries
   c) ARUDINOPATH/libraries

So putting a library inside ARDUINOPATH/hardware/VENDOR/ARCH/libraries works. Again, here there is another compromise to deal with: we can reorder the path priority for example b) c) a), but this won't let the user the possibility to override vendor-provided libraries.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 24, 2013, 07:18 pm
Thanks for taking the time to look at these issues.

Regarding the EEPROM.h, I'm afraid I oversimplified things a bit.  I'm also using inline functions that call C code from within the class definition.  I don't actually have any C++ code for EEPROM on Teensy 3.0, except in that header.  The rest is all C-only, for a number of other reasons I won't mention, for the sake of keeping this reply shorter.

I really do appreciate your offer to merge a pull request quickly.  But with 1.5.5 only days from release, I would really prefer to target 1.5.6 and allow more time to carefully consider these complex issues.  The last thing I want to do is hastily prepare a pull request and cause breakage in an Arduino release, even if it is labeled beta.

It's tempting to write more regarding the parts of this spec that give my the biggest cause for concern, "core-dependencies=" and "architecture=", but at this moment I'd really prefer to think about these a while longer before more comments.  Likewise for customizing the examples.

The library search order is one place where I'd like to make a proposal.

Quote

The current IDE should work, more or less, like this:

Let's call ARCH the architecture of currently selected board, and VENDOR its vendor name.
1) The IDE discards all 1.5 libraries that haven't ARCH in the platform= property.
2) From the remaining libraries it choose the first matching library by searching these folders, in order:
   a) SKETCHBOOK/libraries
   b) ARDUINOPATH/hardware/VENDOR/ARCH/libraries
   c) ARUDINOPATH/libraries

So putting a library inside ARDUINOPATH/hardware/VENDOR/ARCH/libraries works. Again, here there is another compromise to deal with: we can reorder the path priority for example b) c) a), but this won't let the user the possibility to override vendor-provided libraries.


I agree, a) b) c) is the best search order.

However, I would like to propose a smart strategy for the a) to b) transition.  If the user installs a library in their sketchbook, perhaps with "architecture=avr" or any other library.properties line that doesn't match the board they're selected, then b) and c) should be tried to find a better match.  Libraries found in b) should always match, and c) is the last location, so really the smart decisions are only truly needed at step a).
Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 24, 2013, 09:29 pm
Cristian,
You did answer one thing that I thought was probably the case:
The "architecture" names are global across all vendors.

As far as my question goes about how a 3rd party library can specify which core to use
I think you have missed part of the issue.
What I was questioning was more how does a library specify that it
only works with certain specific cores?
For example, suppose the library only works on a specific vendors core
and none of the others - even if they are the same architecture.
i.e. maybe something only works on the pjrc teensy avr core.

Then there is the case where a library only works on a certain chip.
Maybe it only works on a board that has a 32u4 on it
or maybe it only works on board with a m168 or m328.
Libraries only working on specific chips is quite common.
This common real-world scenario can't be handled by the current .properties spec.

What I'm saying is that I wasn't concerned that the spec could get
a library to work when the user selected a board that supported it even when it was
on a 3rd party core or 3rd party board.
It was more that the there is not enough information to prevent the user from trying to use
it on boards that can not support it.
It was my assumption that this was a goal of the .properties stuff in the first place.
To provide enough information to allow the IDE to filter out and not show
libraries and examples that will not work on the board the user has selected.
And if we going to allow the user to pick libraries and examples that won't work
on and/or error off during compilation for the selected board,
we are missing out on what could be done because that is essentially what we have today.

And then what about sketches?
Sometimes users go off and download sketches.
Currently there is no way for a sketch author to specify which cores or libraries may
be required other than with comments in the code.
We see many users posting questions in the forums asking why the sketch or example
that they download isn't compiling because of missing libraries or it not working
on the board that they have.


I guess what bothers me is that the current spec looks like it is only capable of
solving some of the issues while ignoring several current real world issues,  and I don't really like partial solutions.


With respect to "cores" and core-dependencies,
I think I now see the problem.
The Arduino team point of view seems to be that the environment as fairly consistent
for a given release.
And for them this is case because they ship a full blob that is all done at the same time and is all the same
level/version. The Arduino team really doesn't have to deal with past releases in their libraries.
However, once you enter into the 3rd party world, this is no longer the case.
In the 3rd party world, a core or library is done and while the Arduino IDE and its cores advance
the 3rd party library and core may not be updated right away or at all.
Also, a 3rd party library and core cannot always depend on only using the latest Arduino release
the way the Arduino team can with its libraries.
Many 3rd party libraries today are capable of running on pre 1.x as well as 1.x
and even 1.5 (by using appropriate ifdefs).
And this is why backward compatibility is so important to the 3rd party world,
whereas to the Arduino team they have no need for backward compatibility
because they essentially re-do everything all at once on each of their releases
and get to leave old releases behind.
This is why the 1.0 library change was no big deal to them, while it was a HUGE deal
the 3rd party developers.

So back to why this matters for the "core-dependencies".
After seeing the response to Paul's question it appears that
there really is no vendor/maintainer core dependency in the spec at all.
The "core-dependencies" is actually a version of a architecture core within all the vendors cores.
There appears to be an assumption that all the cores of a given architecture will/must be at
the same version.

I had asked about  about what the "core-dependencies" really meant in previous posts.
My assumption had been that it referred to the vendor/maintainer to allow the library
to specify which vendor's core it worked on.
But now it appears that this is really a revision of the "arduino" core within all the architectures.
Note: The naming seems VERY confusing in that there is a "arduino" vendor
and subdirectory in the hardware directory and now we have another "arduino" name
that means something completely different.
What really confuses me is that the library spec says:
Quote
core-dependecies - the core on which this library works on (multiple cores can be specified)

So my assumption was "cores" like Teensy vs ManiacBug vs Arduino etc...
But this seems to not be the case.
It appears that the "core-dependencies" values are referring to the revision
of an architecture core.

While I can see the use for the ability of the library to ensure that a particular "arduino" core API
is supported within a given architecture core, my concern is that it appears to be tracking the
IDE revisions because that's how thins are released for the Arduino team code.
This concerns be greatly.
It also does not appear to allow using or requiring different versions for the different architectures.

Consider this case:
Down the road the "arduino" API advances and you have a situation
with multiple maintainers for a library.
For example,
the latest "arduino" API is now 1.5.9
The library previously supported an avr architecture and works on any API back to 1.5.0
Now support is being added for sam but the sam developer decided to use features
from 1.5.9
So now the sam code requires >=1.5.9 but the avr code can run on >=1.5.0
There does not appear to be a way to specify that.

This is why earlier I brought up the possibility of specifying the core version
on the architecture= values.
It allows the library to run with mismatched versions on different architectures
as well

So you could have:
architectures = vendor/architecture/core (version)

This would support most of the capabilities I've been talking about
including the ability to specify individual vendors cores.
(It wouldn't handle the library only working a specify chip).

So for the above example:
architectures=*/avr/arduino (>=1.5.0), */sam/arduno(>=1.5.9)

which is any vendor with "avr"  architecture  "arduino" core and then
the desired/needed revisions of that core.

For a library that only works on Pauls teensy avr core:

architectures=pjrc/avr/arduino

For a library that only works on maniacbug
architectures=maniacbug1284p/avr/arduino

For a library that works on any "arduino" core >= 1.5
architectures: */*/arduino (>= 1.5.0)
(any vendor, any architecture, but requires arduino core >= 1.5.0)

To me this seems to make more sense than the current
core-dependencies since I simply can't resolve how that really
works once you have more than one version and multiple architectures.

etc...

or perhaps some sort of hierarchical structure like what is in
the platform and board files to allow the specific architecture of
board preferences override the more general settings.


For sketches, I think it would seem logical to use the same
type of .properties file.
The sketch would have its library dependencies,
and those libraries may have additional dependencies
to form the final dependencies for the sketch.



All that said, I think that there needs to be more thought to come
up with a way that allows a library to specify that it only works on
a specific chip.

Perhaps using the property values like what is in the board and platform
files is the answer.

--- bill




Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 24, 2013, 10:08 pm
Bill, precise fine-grain compatibility specs would be excellent on a technical level, but would they also work well on a human level?

A scenario that concerns me looks something like this:

1: User A writes code to interface some sensor to Arduino Uno.
2: Forum users B, C, & D request the code, so he packages it into a library and uses a library.properties for Uno.  User A posts the code to a forum topic.
3: Leonardo users can't use the library, even though it would work, only because library.properties is too narrowly specific.
4: User C with a Leonardo adds the specs for Leonardo, posts an updated copy later on the same forum topic.
5: User B has an Arduino Yun, which could run the library, except user C only owns a Leonardo board, and didn't add/test support for Yun.
6: Several months later, Arduino Tre is released.  The library could run on Tre, but can't, only because the spec is again too narrow.
7: Later, someone wants to run it on Teensy 2.0.  I make a modification for Teensy 2.0 and publish a copy in my Teensyduino installer.
8: Yet more users want to run the library on Arduino Micro, Sparkfun's clones, or other derivitives.  Maybe they find the original Uno-only code, or the 2nd port, or perhaps my version.

I could go on and on with this list.....

However, my biggest concern is how code gets ported and adapted to different boards.  Real users simply try using a library on an unsupported board.  They get errors and ask for help on the forum.  Hopefully they post the actual error messages.  The "copy error" button in 1.5.X is intended to gently guide users to post actual errors.  Expert forum users see the error and suggest fixes.  Sometimes the original author updates their library.  Sometimes abandoned libraries get picked up by others (really, I believe the most beautiful thing about open source).

My point is the "just try it and see what happens" approach, messy and error-prone, is a pretty important mechanism by which the Arduino community collaborates.  Many people who know a lot more about human interaction than me have written about the importance of empowering people to try things and accept/learn from setbacks and failures.

Yes, some libraries (notably ones you've written) are done with an expert approach, where one developer pours tremendous effort into cross-platform compatibility and verification.  But most libraries aren't developed that way.

An air-tight system that prevents users from trying to compile code on specific boards the author or maintainer hasn't certified as compatible might spare users from compile errors, but it might also have a pretty negative overall impact on community collaboration and sharing.

Title: Re: Comments on Arduino 1.5 Specifications
Post by: bperrybap on Oct 24, 2013, 10:24 pm
Paul,
It isn't possible to have it both ways.
Either the system allows specifying which cores/architectures/boards/chips and other libraries
it works on or it doesn't.
I'm ok with either way.
The problem is having a half way in the middle type of approach which is the way
I view the current .properties spec.

I guess it comes down to what is the point/purpose of all
this .properties stuff?
It doesn't seem to be obvious from looking at the spec or looking
at the behavior of the 1.5.4 IDE.

My assumption was that it was to ensure that all needed resources existed
to support a library. Those resources include, other libraries, cores, and hardware.

If it isn't to prevent a sketch/library from being compiled on non supported h/w
and s/w then I think there are MUCH simpler solutions to this library properties stuff.


--- bill
Title: Re: Comments on Arduino 1.5 Specifications
Post by: pjrc on Oct 24, 2013, 10:44 pm
I really wish I would have put more thought into this when I first saw it several months ago.

The more I think about it now, the more concerned I feel it may create artificial barriers to humans collaborating.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: cmaglie on Oct 25, 2013, 12:10 am
Guys,

everyone of you made great points here.
I would ask you to continue the discussion on the developers mailing list, that is where these discussions should take place.

Please give a concise and clear recap of your concerns, and I'm sure that we can come up to a great library specification.
Title: Re: Comments on Arduino 1.5 Specifications
Post by: avenue33 on May 19, 2014, 01:19 pm

Guys,

everyone of you made great points here.
I would ask you to continue the discussion on the developers mailing list, that is where these discussions should take place.

Please give a concise and clear recap of your concerns, and I'm sure that we can come up to a great library specification.


Sorry, too late. I've switched for the LaunchPad platform from Texas Instruments: cheaper, better, stronger.

There's the same Processing-based IDE called Energia (http://energia.nu) which is compatible with the Wiring / Arduino framework. So I can reuse all my projects with no or minimal adaptation. Moreover, Energia manages the MSP430 and TM4C platforms in a very clean way.

The LaunchPad Tiva C TM4C123 (http://embeddedcomputing.weebly.com/launchpad-stellaris-lm4f120.html) is a real bargain at US$13 with 256 KB flash, 32 KB RAM, 80 MHz ARM M4 with FPU, 40 pins and an integrated programmer/debugger. The Connected LaunchPad Tiva C Series TM4C129 (http://embeddedcomputing.weebly.com/connected-launchpad-tiva-c-series-tm4c129.html) goes even further: for US$20, 1 MB of flash, 256 KB of RAM, 120 MHz ARM M4 with FPU, 98 pins, built-in Ethernet and an integrated programmer/debugger.

There's the same vibrant hobbyists community (43oh (http://forum.43oh.com/forum/28-energia/) and Stellarisiti (http://forum.stellarisiti.com/forum/63-energia/) forums) but also professional support at the E2E engineer-to-engineer (http://e2e.ti.com/support/microcontrollers/default.aspx) forum hosted by Texas Instruments.

Of course, I'm using embedXcode (http://embedxcode.weebly.com) :)