Reputation system for libraries

Not sure where to put this, but it seems like something we need. Seems like I'm doing a lot on ASE and SO that ends up having to do with someone having a totally crap library.

I get it. When noobs learn about classes and libraries they immediately think, "hey I can make it so nobody ever has to learn about millis" and writes their own light blinking class. It's a GREAT exercise in learning to code and I don't want to discourage that at all.

But what happens is that they get all proud of their contribution and think it's the great new thing and then they release it and it gets into the ecosystem and others pick it up and try to use it. Pretty soon were seeing people having problems they can't even blink a light and what it boils down to is that they are using a library that was built by a noob as a learning exercise that is still completely full of bugs.

Could there be some sort of reputation system out there for libraries? Maybe a way to vote them up or down. I'm not sure how it could work, but it would be really nice for the raw green noobs who have no idea how to tell the good from the bad to at least have some sort of "seal-of-approval" that they can look to and know that they're getting something that works and not someone's practice attempt.

Just a thought.

I think the closest thing to an existing system is GitHub. You can get a general feel for the library just by checking the stats on its home page (stars, watchers, forks, contributors). If you take a look at the issue and pull request tracker, you get an even better idea. Then the commit history is my next stop. Just by looking at the commit messages, I can get a pretty good idea of the quality of the development work. If you see a lot of those automatically generated commit messages GitHub does when someone is too lazy or clueless to write one, that's a huge red flag.

Other than the commit messages, the other indicators have the issue of taking time to accumulate, so a high quality library won't look very good when it's new. However, I think any crowd sourced solution will have the same problem.

The big advantage of GitHub is that the system is already well established and 99% of Arduino libraries are already on there. It's doubtful that anything one of the forum members created could gain that much traction. The system will only work if people take the time to "star" the libraries they have found to be of good quality. As for the "down vote', I suppose the GitHub equivalent would be opening a bug report or pull request.

As for a "seal of approval", the closest equivalent on GitHub would be a passing CI build. I've done, and continue to do, a lot of work to make it easy for people to set up automated tests of Arduino projects. I would love to see more Arduino projects use CI. I'm always amazed to find that an apparently competent developer didn't bother to click the "compile" button before making a release. Of course, a compilation test can't tell you whether the code actually works, but it's a lot better than nothing. Presence of a working CI system with a passing build is another indicator of quality. The next step would be unit tests, but I haven't seen much of that happening in the Arduino world.

You could build something to aggregate the information from the Arduino Library Manager index and the GitHub API to present this information in some other form. An example of this is arduinolibraries.info.

I wrote a script that finds all the Arduino libraries on GitHub and scans them for common issues:

Some of the things it checks for are pretty minor (e.g., invalid keywords.txt formatting), but they could be considered an indicator of quality. After all, if a library author won't take the time to read and follow the well-documented specification for keywords.txt, and doesn't check whether keyword highlighting is working, what does that say about their code? That project also checks for some more serious issues that will definitely give a beginner a bad day. It also gets the status of the default branch's CI build, which tells you whether there is a CI system or not, and whether the checks are passing. So perhaps some additional checks like this could be integrated into an website that aggregates all the available quality indicators in one place to automate the evaluation process as much as possible.

I don't know what could be done to automatically evaluate code. I know there are linters for C++ and C, but I haven't seen these used much in the Arduino world and my quick attempts were not fruitful. You could do a compilation test, but how do you know which boards to compile for. The Mouse library says it's for AVR, but if you compile the example sketches for all the AVR boards, you'd find it failed most of the compilations, not because there is anything wrong with the library, but because only specific boards are compatible. I haven't figured out any way to make this work other than someone setting up the CI build for the project. It would be a great project for someone to submit PRs for CI builds for the top n Arduino libraries that don't already have CI set up.