[arduino 1.5] "Not a valid library" importing Bitlash

Hello:

I am testing the brand new Arduino 1.5 beta here on OS X. When I try to import the Bitlash library into 1.5, I get "Not a valid library". Anyone know why, and how I can fix it?

You can get Bitlash here if it's needed for testing: GitHub - billroy/bitlash: Bitlash: a programmable command shell for arduino

Update: related bug: after the "Not a valid library" message appears, the "Import Library..." item is gone from the Sketch menu.

Update: Other libraries which appear not to import to this version, in my configuration: AFMotor, ArdOSC, Fat16, NewSoftSerial, RF22, and SDFat.

I can't find the phrase "Not a valid library" anywhere in the Arduino source code on Github.

-br

Update: The AFMotor and NewSoftSerial libraries can be fixed by renaming the "Examples" folder they contain to "examples". After restart they are recognized and appear in the IDE menu.

All the libraries which fail, including Bitlash, seem to have additional subdirectories besides "examples" in them.

I've opened a bug: Google Code Archive - Long-term storage for Google Code Project Hosting.

-br

Thanks billroy,

the issue with "Example" folder with capital E will be fixed on the next beta release.

about the other libraries, you're right: libraries can't use folders (except for 'example' and 'utilities').This is due to the way the 1.5 IDE support "multiplatform" libraries. Probably its better to disable such "feature" on the next beta release until further discussions about the multiplatform library format happens.

C

cmaglie:
about the other libraries, you're right: libraries can't use folders (except for 'example' and 'utilities').This is due to the way the 1.5 IDE support "multiplatform" libraries. Probably its better to disable such "feature" on the next beta release until further discussions about the multiplatform library format happens.

That is going to give some compatibility issues. Looking at the libraries I have installed (which isn't a huge number) and excluding exact matches to 'example' and 'utilities') :

  • Ardupilot libraries have 'examples', 'include', 'tools'
  • ArdOSC has 'Example' and 'OSCCommon'
  • Encoder has 'util'
  • Ethernet has 'utility'
  • FreqCount and FreqMeasure have 'util'
  • ks0108 has 'Processing/glcdBitmap/data'
  • LedControl has 'doc'
  • MIDI has 'doc/html'
  • SD has 'utility'
  • Timer1 and Timer3 have 'config'

Yes I think disabling folders is a bad idea, we all like when things are organized in folders :slight_smile:

Well I think libraries will have to be adapted anyway to work on the new Due.
there is a new format for "Fat" libraries that supports multiple versions in the same library and having some rules on how subfolders can be called is going to help make libraries more portable.

many of the libraries mentioned in this thread make low level access to the AVR registers, obviously these are not going to work on the Due and will require modifications.

m

Massimo, I have the highest respect for your judgement and this platform. It's your ecosystem. You can break it if you want to.

But hear me out. As a long-time Arduino library developer, I can tell you it's both frustrating and disheartening to have one's work broken without warning, again and again.

Massimo: You won. This platform is a huge accomplishment. There are existing projects on Duemilanove boards that will be in operation for decades, perhaps long after we are dead. This is goodness. But unless the Arduino team perceives it to be part of its mission to provide a tool chain that will be compatible over that kind of time scale, those systems will become useless the first time a bug is found and an old library needs to be recompiled to fix it.

I would agree library developers have a responsibility to track the core. But it is a fact of life that there are libraries that are not going to be maintained any more and therefore need to be buildable in their present state.

The thing that is particularly egregious about this proposed change is that it not only breaks current libraries, but it also prevents co-existence of future versions with earlier versions of Arduino needed to build older projects.

From my perspective, the team should take more seriously the impact of its decisions to break backwards compatibility. If there is a way to keep the old libraries working, then keep them working, even if it's ugly compared to the shiny new stuff. In my view, this is a responsibility that comes with the success of this project.

If the core team does not agree with this view, it would be helpful to hear.

Thank you for listening. I'd be happy to follow up either here or privately.

Best regards,

-br

Again, requiring modifications to work and requiring removal of useful information to a different place for no obvious reason are two separate things.

I have to agree 100% with bilroy on this.

billroy:
Massimo, I have the highest respect for your judgement and this platform. It's your ecosystem. You can break it if you want to.

But hear me out. As a long-time Arduino library developer, I can tell you it's both frustrating and disheartening to have one's work broken without warning, again and again.

Massimo: You won. This platform is a huge accomplishment. There are existing projects on Duemilanove boards that will be in operation for decades, perhaps long after we are dead. This is goodness. But unless the Arduino team perceives it to be part of its mission to provide a tool chain that will be compatible over that kind of time scale, those systems will become useless the first time a bug is found and an old library needs to be recompiled to fix it.

I would agree library developers have a responsibility to track the core. But it is a fact of life that there are libraries that are not going to be maintained any more and therefore need to be buildable in their present state.

The thing that is particularly egregious about this proposed change is that it not only breaks current libraries, but it also prevents co-existence of future versions with earlier versions of Arduino needed to build older projects.

From my perspective, the team should take more seriously the impact of its decisions to break backwards compatibility. If there is a way to keep the old libraries working, then keep them working, even if it's ugly compared to the shiny new stuff. In my view, this is a responsibility that comes with the success of this project.

If the core team does not agree with this view, it would be helpful to hear.

Thank you for listening. I'd be happy to follow up either here or privately.

Best regards,

-br
http://bitlash.net

Wise words.

billroy:
Massimo, I have the highest respect for your judgement and this platform. It's your ecosystem. You can break it if you want to.

But hear me out. As a long-time Arduino library developer, I can tell you it's both frustrating and disheartening to have one's work broken without warning, again and again.

Massimo: You won. This platform is a huge accomplishment. There are existing projects on Duemilanove boards that will be in operation for decades, perhaps long after we are dead. This is goodness. But unless the Arduino team perceives it to be part of its mission to provide a tool chain that will be compatible over that kind of time scale, those systems will become useless the first time a bug is found and an old library needs to be recompiled to fix it.

I would agree library developers have a responsibility to track the core. But it is a fact of life that there are libraries that are not going to be maintained any more and therefore need to be buildable in their present state.

The thing that is particularly egregious about this proposed change is that it not only breaks current libraries, but it also prevents co-existence of future versions with earlier versions of Arduino needed to build older projects.

From my perspective, the team should take more seriously the impact of its decisions to break backwards compatibility. If there is a way to keep the old libraries working, then keep them working, even if it's ugly compared to the shiny new stuff. In my view, this is a responsibility that comes with the success of this project.

If the core team does not agree with this view, it would be helpful to hear.

Thank you for listening. I'd be happy to follow up either here or privately.

Best regards,

-br
http://bitlash.net

@billroy

The whole purpose of having a separate IDE for a while is to work with the community to estailish standards to use in the future.
Adding a new core to the IDE to support a completely different processor is going to bring pain, we have done some preparatory work to be able to do that, the work is clearly not complete.

I don't exactly understand when you say "to have one's work broken without warning, again and again."
In 2010 we had a meeting with a number of prominent members of the community, we even flew a few of them from japan and europe to NYC to discuss the future Arduino 1.0, we blogged about it and we discussed it on the developer mailing list.
We mentioned over and over that we were going to break a few things in order to have an 1.0 API that could stay stable for a long time.
We released beta IDE's for a while and yet when 1.0 came out a lot of people acted like they were taken by surprise.

Now we're bringing to the table a completely new processor with many more possibilities, adding these features to the core is going require that we make some additions to the API keeping the old API stable as much as possible (for example pinMode on ARM should have many more modes, but the old one will still behave as expected.)

We've released the 1.5 IDE as beta, there are bugs, this is obvious. the Examples vs examples is clearly a bug that has been fixed already.

Having said this it's not a bad idea to have a discussion on how to clean up the directory structure of libreries to make them more standard.
At the end of this process we can write a nice wiki page with all the detailed specs and everybody will have the right info to work from.

Change happens , let's work together to drive where this is going. The beta of 1.5 is a chance for arduino to move to a more open development model

PS:where would be the right place to keep in touch with library writer? as it seems that our current channels do not reach all of them.

m

billroy:
Massimo, I have the highest respect for your judgement and this platform. It's your ecosystem. You can break it if you want to.

But hear me out. As a long-time Arduino library developer, I can tell you it's both frustrating and disheartening to have one's work broken without warning, again and again.

Massimo: You won. This platform is a huge accomplishment. There are existing projects on Duemilanove boards that will be in operation for decades, perhaps long after we are dead. This is goodness. But unless the Arduino team perceives it to be part of its mission to provide a tool chain that will be compatible over that kind of time scale, those systems will become useless the first time a bug is found and an old library needs to be recompiled to fix it.

I would agree library developers have a responsibility to track the core. But it is a fact of life that there are libraries that are not going to be maintained any more and therefore need to be buildable in their present state.

The thing that is particularly egregious about this proposed change is that it not only breaks current libraries, but it also prevents co-existence of future versions with earlier versions of Arduino needed to build older projects.

From my perspective, the team should take more seriously the impact of its decisions to break backwards compatibility. If there is a way to keep the old libraries working, then keep them working, even if it's ugly compared to the shiny new stuff. In my view, this is a responsibility that comes with the success of this project.

If the core team does not agree with this view, it would be helpful to hear.

Thank you for listening. I'd be happy to follow up either here or privately.

Best regards,

-br
http://bitlash.net

Massimo:

Thanks very much for your reply. I invite you to stop by next time you are in Denver to hear about Arduino development from this library developer's point of view. Maybe we can get Nate to pay for the beer and sell tickets.

You don't address my point about breaking backwards compatibility and creating maintenance headaches for your best customers who have a lot of old projects to maintain.

It's really about who you are as a business and an institution. Do you agree or not with this statement: "If it's possible to keep existing Arduino projects building in the IDE, it should be done!"

If the core team accepts that as a requirement, the conversation is over and we can all go back to coding.

In my opinion, if the core team doesn't accept that as a requirement, it changes the whole value proposition of the Arduino ecosystem. How do I pitch Arduino when it's reasonable to ask whether the tool chain, including the third party libraries we all use, will still support the project when it's in maintenance a couple years down the road?

Long term support is an evolving thing in open source. Massimo, I'm asking you and your company to help me help our shared users to have a better experience in the long run with this platform by avoiding this obvious and preventable mistake. Please listen.

-br

So I have a question that this topic has seemed to have brought up. Not that I have or may ever release a 3rd party contributed library, but if I was and only developed it and tested in on AVR boards, would I be obligated to write it such that it could handle the new ARM based Due board? Would just having comments in the start of the .h file telling which processor(s) it was designed to run under be enough. What would be the proper etiquette for 3rd party library developers to use when releasing new library submissions in the future when it comes to what boards the code can support?

Again I'm more a hardware guy then software guy, but ever sense the announcement of the Due board I've felt that if would be best not to try and support both AVR and ARM based boards in the same IDE, but rather have separate IDEs that don't try or have to be 'compatible' with each other, they could certainly have the same look and feel, but be totally separate platforms. I just think there is a real possibility of taking on more then can be delivered and end up losing the simple to learn and use IDE that has been so useful for newcomers to the platform?

Lefty

The new IDE 1.5 has 'fat' libraries support: libraries that contains code to run on more than one microcontroller architecture. The advantages of having a 'fat' library are clear:

  • The IDE will automatically lists the available libraries based on the architecture of the currently selected board
  • The same library can have multiple architecture inside => you don't have to install a different version of the same library for every architecture.

We also tried to introduce an auto-detection mechanism that takes apart non-fat libraries from fat libraries.
This mechanism didn't worked as expected because we assumed that library writers would only use 'examples' and 'utility' folder. This is a clearly wrong assumption, and the patch i have just pushed on the repository fixes this problem.

So, guys, don't worry: all your libraries will continue to work as expected (of course if you try an AVR specific library on the Due it will not work).

I'll invite you all to check if the patch will fix the issues you are having.

C

Thanks, C. Tested and working here for Bitlash and the other libraries I had problems with.

For others who may want to test, here's what it takes to fire up the build with these changes (OS X, Xcode tools installed):

> git clone https://github.com/arduino/Arduino.git
> cd Arduino
> git fetch origin ide-1.5.x
> git checkout ide-1.5.x
> cd build
> ant
> ant run

I love the sound of "libraries will continue to work as expected." Thanks.

Now to go chase some compiler errors on the Due side...

Question for C: is there an official #define that distinguishes the Due/Arm build from the AVR builds?

-br

I've pushed an update to Bitlash that compiles and links cleanly with the ARM processor selections using the current github ide-1.5.x branch of the IDE. It has a 4k simulated EEPROM situated in ram for function storage, for now, so your functions aren't saved through power off.

I don't have a Due handy, so I can't test it, but I'd love to hear field reports.

-br

billroy:
Question for C: is there an official #define that distinguishes the Due/Arm build from the AVR builds?

He's already covered that in this thread:

http://arduino.cc/forum/index.php/topic,128520.0.html

cmaglie:
The new IDE 1.5 has 'fat' libraries support: libraries that contains code to run on more than one microcontroller architecture. The advantages of having a 'fat' library are clear:

  • The IDE will automatically lists the available libraries based on the architecture of the currently selected board
  • The same library can have multiple architecture inside => you don't have to install a different version of the same library for every architecture.

These are good changes.

cmaglie:
We also tried to introduce an auto-detection mechanism that takes apart non-fat libraries from fat libraries.
This mechanism didn't worked as expected because we assumed that library writers would only use 'examples' and 'utility' folder. This is a clearly wrong assumption, and the patch i have just pushed on the repository fixes this problem.

Pleased to hear that the feedback has been taken int account.

To make my position clear, some sort of cleanup by documented standard directory names is a good thing. Consolidating 'Example' and 'example' and 'examples' and suchlike variants into a single, documented name is a good thing (so the IDE can display te eample,s, in a menu).

Likewise for other directories on which the IDE has to automatically act. I could see integrated documentaton browsing, for example, if tyhere were standard ways to add it (doxygen seems to be popular). Keeping documentation close to code, and encouraging documentation, are clearly good thins.

On the other hand there may be a need for all sorts of other data, the form of which may ary wildly depending on the purpose of the library and thhe supported hardware. Libraries should be free to add what they need, since the IDE does not need to know about it or do anything with it. Same for setup utilities, python and perl scripts to generate config files or to consume data produced by the library, etc. Although having documented standard values ('data' and 'utils' for example) would help consistency.

cmaglie:
I'll invite you all to check if the patch will fix the issues you are having.

GitHub - arduino/Arduino at ide-1.5.x

Yay. Checking.

Is the RAM abailable to Bitlash configurable? At boot on my DUE bitlash says it only has 1000bytes free.