Running automated unit tests on an Arduino library using Travis CI

I'm pleased to announce a system for unit testing Arduino libraries capable of supporting continuous integration (CI) on GitHub / Travis CI. The purpose of this is to make it easier to accept code contributions to your Arduino library on GitHub from complete strangers -- you don't have to bother downloading / compiling / uploading their code in order to test it manually.

Instead, you let the computer do the testing for you.

In addition to the writeup I did on StackOverflow, the project README is fairly up to date and I have a few working examples (Arduino-Queue, Adafruit FONA) to show it in action.

Is this sort of project relevant for this community, or are things like GitHub collaboration and Continuous Integration beyond the scope of practical Arduino work?

More thorough and easier testing of Arduino projects will lead to less bugs, the developers being able to focus on the authors being able to make improvements rather than spending their time on testing and addressing bug reports, and will generally make things easier for all the people providing support to Arduino users. For these reasons I think your project is a great contribution to the Arduino community. Thanks!

I've been spending a lot of time getting familiar with the use of Travis CI on Arduino projects and have been trying to help add this capability to some other Arduino projects. I have not yet started my journey into unit tests but that is definitely on my to-do list after I've reached a more finished stage in my current CI projects. So I haven't yet used your project but I do have a couple of comments based solely on my reading of your documentation. Forgive me if I have missed something obvious.

I notice that your system expects the unit tests to be located under the test subfolder of the library. The Arduino Library Specification states:

An extras folder can be used by the developer to put documentation or other items to be bundled with the library.
...
The content of the extras folder is totally ignored by the IDE; you are free to put anything inside such as supporting documentation, etc.

So it seems like extras/test would be the recommended location for internal testing files to be placed.

One place where automated testing is especially useful is for Arduino hardware packages. Typically these will include architecture specific libraries and testing these libraries against the various board configurations of the package can be extremely time consuming if done manually. If your project supports it, I think it would be helpful to add a demonstration of that usage.

I'm not sure how best to handle the "extras" folder. On one hand, it would be more consistent with the documentation as you pointed out. On the other hand, it would make the definitions of the tests a bit more kludgy by placing them further from the code that they're testing. And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously and just have the IDE consider the "test" folder to be a normal and welcome addition to a project.

I don't expect that to happen in the foreseeable future though.

Can you give some examples of what you mean by a hardware package? I do have some capabilities to define compiler flags and custom platforms to test on, but I'm not sure whether that relates to what you're talking about. Do you have a link to the library in question?

Here are some popular examples of what I call "hardware packages" (also known as "cores"):

Hmm, if I'm reading this right then you're talking about running hardware tests on the board itself. I don't think I can do that -- I strip out all the hardware-related functionality and provide software substitutes for it. Only the board constants get pulled in (#defines for comm port pins, those sort of things).

I'm not talking about running tests on the physical hardware. I'm talking about doing unit tests of the libraries bundled with the hardware packages. For example:

Aah! Yes, this should be possible. But possible-ish... I'm not sure how I'd get the cores integrated with the codebase. I had to pull in a lot of the Arduino header files manually to get things to compile properly:

It's not immediately clear how I'd get a custom set of headers to integrate with that. When you develop such a library, where do you put it? Is it something that could be downloaded as a package from a custom source? I have the ability to do that:

Well the goal would be to install the hardware package as it exists in the current development state. Some hardware packages do have the JSON file for Boards Manager installation but that will install a release version of the package rather than the tip. So, for example, for the compilation tests of the MightyCore bundled library examples sketches via Travis CI the first step is to install the Arduino IDE and then copy the contents of the repository to the hardware subfolder of the sketchbook folder. This is known as the manual installation method, as opposed to the Boards Manager installation method. Then the compilations are done not only with the latest versions of the library code but also the latest version of the hardware package.

I think you sort of answered your own question, you'd need to either configure things locally such that the yml file described what was on your disk, OR, automate the steps to do that as part of the Travis CI build script.

The good news is that my script only installs the Arduino IDE if it can't find it. arduino_ci/arduino_installation.rb at master · Arduino-CI/arduino_ci · GitHub

So this should be workable. Want to try this out?

Just to clarify this a bit: my system lets you define platforms, and platforms are based on packages. This is done in the .arduino-ci.yml file.

If the board/core packages you want are known to your Arduino IDE already (like, they are installed in a local directory that you've already set up), no problem -- define a platform in the yml file that refers to that package. If not, your options are to either specify the package info in the yml file (the URL for them), or to circumvent that requirement by setting the Travis CI script to install the files you need in the same location that they'd exist on your local machine.

Does this make any sense? TL;DR I think it will work.

Yes, it makes perfect sense. I haven't had a chance to try it out yet but will do so ASAP. Thanks.

No rush, I only doubled my reply because I re-read my initial one and realized that it was a very poor answer to your question :slight_smile:

You'll have to excuse my excitement though; this wasn't a feature that I had in mind when I wrote the system, and it makes the extra effort I put into the yaml file layout seem much more worth it. As much as I'm interested in whether it works for you, I'm also interested in knowing all the steps you have to take in order to make it work.

My thinking is that it might be possible for me to add a config entry to the YAML file that would let you specify a path on disk (or some other locator) for a still-in-development hardware package. Then it's just a question of how to act on that information (e.g. forcibly copying it into the Arduino IDE each time, in case changes were made to it). I just need to learn more about the development cycle for hardware packages.

ifreecarve:
And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously

It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

Does the Atmel development system support unit testing?

...R

I'm glad you're excited about it. Work on Arduino hardware packages was what got me initially interested in testing automation because it was just such a huge amount of work just to compile all the possible combinations manually that the job of testing was sometimes ending up getting put on the users. That's fine with willing beta testers but it's a bad experience for unsuspecting beginners who probably assume the bug they ran across is something they are doing wrong.

Generally a hardware package repository will be structured so that it can just be directly copied into the hardware subfolder of the Arduino sketchbook folder for manual installation. The folder structure required for Boards Manager installation is a little bit different in that there is no architecture folder so everything is moved up one folder level. Some repositories do have the Boards Manager installation structure, for example: ESP8266, ESP-32, nRF-52. All three of those require toolchain installation so they would need to go through some extra package installation steps anyway and thus structuring the repo for easy manual installation is not really beneficial.

Robin2:
It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

I do agree that the target Arduino user would only be overwhelmed by the concept of unit testing. However, I think it is reasonably in reach for the average 3rd party Arduino library or hardware package developer, especially if people like ifreecarve help to make it more accessible. I've been noticing compilation tests using services like Travis CI becoming more common in the Arduino world but unit tests are pretty rare. I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss. Being able to check the results of the Travis CI build is a great way for a potential user to do a quick initial evaluation.

I agree that unit testing an Arduino sketch is beyond the market of hobbyists that Arduino is aiming at.

On the other hand,a brief sampling of Arduino projects turns up smoke/fire alarms, automotive projects, heavy equipment control, hydraulics, drones, healthcare, etc. Software errors on projects like these can cause real harm. Obligatory Therac-25 reference goes here.

I don't have a problem with a lack of ability to test individual sketches; each person can take responsibility for their own work. However, the practice of using libraries -- code provided by complete strangers via the Internet -- without an ability to evaluate their quality or correctness, does cross a line into being a Bad Idea for some applications.

We should at least have the option to create tests for library code, regardless of whether the majority of library writers will make use of it.

pert:
I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss.

I have not come across libraries that simply don't work. My experience with libraries and from reading about Forum members using libraries is that there are two main problems

  • Totally inadequate documentation
  • Incompatibility between libraries.

I don't believe unit testing can address either of those problems.

To some extent it may never be possible to eliminate all incompatibilities between libraries simply because there is a very limited set of hardware that they must all share.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

...R

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

I think that's the case for this project, but if you try it out and believe otherwise then I would welcome your feedback.

Robin2:
I have not come across libraries that simply don't work.

I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

Agreed. The GUI that the average user interacts with must be kept very simple. But the Arduino IDE can allow for advanced usage under the hood. Regarding the test folder thing (which is the only specific change that's been mentioned) I'm not sure it would even end up as a change to the current IDE code but only a change in the text of the library specification. Currently the Arduino IDE has a source folder whitelist approach, meaning that there shouldn't be any problem putting the unit tests in the test folder. I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

pert:
I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

Yes, that's accurate. In Java, there would be a "src" directory for the code and "test" directory for the tests. In Python the code would live in a directory that's the same as the library name, and the tests in "test". Ruby would have a "lib" and a "spec" directory, and JavaScript a "src" and "test" directory. In Swift, code would be split among a number of directories but tests live in "Tests". Etc.

I don't think it makes any sense to clutter the Arduino IDE. Adding a whitelist entry for "test" to the library specification would be best-case-scenario for me, including (and/or especially) if they didn't prescribe a particular test tool to manage that. It would be up to the folks who want to do collaborative development via GitHub to implement their own tests and CI in that directory.

pert:
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

I suspect part of the problem is that the authors of libraries don't have access to a wide range of Arduino boards.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?

I think I would be content with decent documentation that, inter alia, lists the boards that the author has tested the library on. And I don't mean unit tests.

If people can't be bothered to write documentation what chance is there that they will take the trouble to learn how to do unit testing.

...R