Running automated unit tests on an Arduino library using Travis CI

Well the goal would be to install the hardware package as it exists in the current development state. Some hardware packages do have the JSON file for Boards Manager installation but that will install a release version of the package rather than the tip. So, for example, for the compilation tests of the MightyCore bundled library examples sketches via Travis CI the first step is to install the Arduino IDE and then copy the contents of the repository to the hardware subfolder of the sketchbook folder. This is known as the manual installation method, as opposed to the Boards Manager installation method. Then the compilations are done not only with the latest versions of the library code but also the latest version of the hardware package.

I think you sort of answered your own question, you'd need to either configure things locally such that the yml file described what was on your disk, OR, automate the steps to do that as part of the Travis CI build script.

The good news is that my script only installs the Arduino IDE if it can't find it. arduino_ci/arduino_installation.rb at master · Arduino-CI/arduino_ci · GitHub

So this should be workable. Want to try this out?

Just to clarify this a bit: my system lets you define platforms, and platforms are based on packages. This is done in the .arduino-ci.yml file.

If the board/core packages you want are known to your Arduino IDE already (like, they are installed in a local directory that you've already set up), no problem -- define a platform in the yml file that refers to that package. If not, your options are to either specify the package info in the yml file (the URL for them), or to circumvent that requirement by setting the Travis CI script to install the files you need in the same location that they'd exist on your local machine.

Does this make any sense? TL;DR I think it will work.

Yes, it makes perfect sense. I haven't had a chance to try it out yet but will do so ASAP. Thanks.

No rush, I only doubled my reply because I re-read my initial one and realized that it was a very poor answer to your question :slight_smile:

You'll have to excuse my excitement though; this wasn't a feature that I had in mind when I wrote the system, and it makes the extra effort I put into the yaml file layout seem much more worth it. As much as I'm interested in whether it works for you, I'm also interested in knowing all the steps you have to take in order to make it work.

My thinking is that it might be possible for me to add a config entry to the YAML file that would let you specify a path on disk (or some other locator) for a still-in-development hardware package. Then it's just a question of how to act on that information (e.g. forcibly copying it into the Arduino IDE each time, in case changes were made to it). I just need to learn more about the development cycle for hardware packages.

ifreecarve:
And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously

It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

Does the Atmel development system support unit testing?

...R

I'm glad you're excited about it. Work on Arduino hardware packages was what got me initially interested in testing automation because it was just such a huge amount of work just to compile all the possible combinations manually that the job of testing was sometimes ending up getting put on the users. That's fine with willing beta testers but it's a bad experience for unsuspecting beginners who probably assume the bug they ran across is something they are doing wrong.

Generally a hardware package repository will be structured so that it can just be directly copied into the hardware subfolder of the Arduino sketchbook folder for manual installation. The folder structure required for Boards Manager installation is a little bit different in that there is no architecture folder so everything is moved up one folder level. Some repositories do have the Boards Manager installation structure, for example: ESP8266, ESP-32, nRF-52. All three of those require toolchain installation so they would need to go through some extra package installation steps anyway and thus structuring the repo for easy manual installation is not really beneficial.

Robin2:
It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

I do agree that the target Arduino user would only be overwhelmed by the concept of unit testing. However, I think it is reasonably in reach for the average 3rd party Arduino library or hardware package developer, especially if people like ifreecarve help to make it more accessible. I've been noticing compilation tests using services like Travis CI becoming more common in the Arduino world but unit tests are pretty rare. I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss. Being able to check the results of the Travis CI build is a great way for a potential user to do a quick initial evaluation.

I agree that unit testing an Arduino sketch is beyond the market of hobbyists that Arduino is aiming at.

On the other hand,a brief sampling of Arduino projects turns up smoke/fire alarms, automotive projects, heavy equipment control, hydraulics, drones, healthcare, etc. Software errors on projects like these can cause real harm. Obligatory Therac-25 reference goes here.

I don't have a problem with a lack of ability to test individual sketches; each person can take responsibility for their own work. However, the practice of using libraries -- code provided by complete strangers via the Internet -- without an ability to evaluate their quality or correctness, does cross a line into being a Bad Idea for some applications.

We should at least have the option to create tests for library code, regardless of whether the majority of library writers will make use of it.

pert:
I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss.

I have not come across libraries that simply don't work. My experience with libraries and from reading about Forum members using libraries is that there are two main problems

  • Totally inadequate documentation
  • Incompatibility between libraries.

I don't believe unit testing can address either of those problems.

To some extent it may never be possible to eliminate all incompatibilities between libraries simply because there is a very limited set of hardware that they must all share.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

...R

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

I think that's the case for this project, but if you try it out and believe otherwise then I would welcome your feedback.

Robin2:
I have not come across libraries that simply don't work.

I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

Agreed. The GUI that the average user interacts with must be kept very simple. But the Arduino IDE can allow for advanced usage under the hood. Regarding the test folder thing (which is the only specific change that's been mentioned) I'm not sure it would even end up as a change to the current IDE code but only a change in the text of the library specification. Currently the Arduino IDE has a source folder whitelist approach, meaning that there shouldn't be any problem putting the unit tests in the test folder. I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

pert:
I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

Yes, that's accurate. In Java, there would be a "src" directory for the code and "test" directory for the tests. In Python the code would live in a directory that's the same as the library name, and the tests in "test". Ruby would have a "lib" and a "spec" directory, and JavaScript a "src" and "test" directory. In Swift, code would be split among a number of directories but tests live in "Tests". Etc.

I don't think it makes any sense to clutter the Arduino IDE. Adding a whitelist entry for "test" to the library specification would be best-case-scenario for me, including (and/or especially) if they didn't prescribe a particular test tool to manage that. It would be up to the folks who want to do collaborative development via GitHub to implement their own tests and CI in that directory.

pert:
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

I suspect part of the problem is that the authors of libraries don't have access to a wide range of Arduino boards.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?

I think I would be content with decent documentation that, inter alia, lists the boards that the author has tested the library on. And I don't mean unit tests.

If people can't be bothered to write documentation what chance is there that they will take the trouble to learn how to do unit testing.

...R

Well I think one reason developers don't like writing documentation is because, even though they're sharing it, the code is primarily written for themselves and they think (albeit wrongly) of documentation as being an extra task solely for the benefit of other users. However, as a user of the code, there is an obvious direct benefit of having it be thoroughly and efficiently tested.

Automated testing also makes writing code much more enjoyable because you get to do just the fun, creative part that humans are best at and leave all the boring stuff that machines are best at to the computer.

The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way. In fact that's how I've been with unit testing. I've been thinking about it a lot lately but there hasn't been a clear path to actually getting going with it. Maybe arduino_ci will help me finally get there.

I've been working on making it easier to do continuous integration compilation tests of Arduino projects with the hopes that if that initial barrier is lowered then more people will start doing it. So far I don't think I've made any difference but I'll keep working at it.

pert:
The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way.

I don't reckon I do enough Arduino programming to get the pay-off that you describe.

Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

So much of a microprocessor program depends on the physical world outside which cannot be tested using software. And building another microprocessor to simulate the real world is fraught with difficulties, even if there was time to do so.

You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.

In theory I can see a lot of value in test-driven-development - building the test before you write the code. In practice I am lazy.

I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.

...R

Robin2:
Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

Yes, my understanding is that it is a good bit of work writing unit tests. It shouldn't be too bad if you write them as you go along but for existing projects it's going to take a real effort. Even so, it's not all or nothing. You can start with the highest priority tests and add coverage gradually over time.

The compilation tests are a much lower hanging fruit. In my case it mostly consists of having Travis CI use the Arduino IDE to compile library examples. If the examples compile then the build passes. If one of them doesn't compile then the build fails. The trouble is that doesn't tell you whether the code actually works, only that it compiles. I found there were still some tricky parts to doing those compilation tests with Travis CI in the way I wanted so I ended up wrapping all that up in a Bash script, which makes it even easier to set up that sort of continuous integration. It saves a report of the results of the compilations for each build, which makes it easy to compare changes between commits, such as checking if the flash or global SRAM memory usage changed. I can also compile with multiple versions of the Arduino IDE for multiple boards to make sure no incompatibilities were introduced.

Robin2:
I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

Sure, but then afterwards you can write a test for it so it will never bite you again.

Robin2:
You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.

Yes, that's the whole idea behind arduino_ci. I'm still trying to wrap my head around it but you can set and read the states of virtual pins.

Robin2:
In practice I am lazy.

Me too, and what a great feeling it is to kick back and watch Travis CI running thousands of compilations for me.

Robin2:
I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.

Yeah, it's especially nice for open source projects where you get random pull requests. The CI tests automatically run on the pull request so you have this initial triage before you even need to take a look at the proposal. If it doesn't build then the person who submitted the PR can look at the build, find the bug, and then update the PR until it passes the build. A repository maintainer still will need to look at it before it can be merged but that initial bug shakedown can save them a lot of time and effort.

pert:
Yeah, it's especially nice for open source projects where you get random pull requests

That's way way beyond my pay grade :slight_smile:

...R

Robin2:
Would it be possible for an author to run tests that verify compatibility with a board that he does not have?

Yes, arduino_ci does this.

In fact, I was able to develop this system without connecting a single piece of Arduino hardware to my laptop. "Compatibility with a board" is nothing more than setting the right set of preprocessor defines for the compiler. This in turn causes various features to be enabled or disabled (e.g. serial ports).

You can validate this for yourself by attempting to run serial comms unit tests on a board that doesn't have a serial port.

Robin2:
So much of a microprocessor program depends on the physical world outside which cannot be tested using software.

Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause. I provide this power, in a feature that I've termed "GODMODE". If there's a physical scenario that GODMODE can't replicate, then I'll be glad to update the library appropriately.

ifreecarve:
Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.

I don't agree with that at all.

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

To be honest, I don't believe using another microprocessor to simulate the external world would be sufficient.

To my mind there is a huge difference between a test system for (say) a program running on a web server where all the input consists of messages from another computer program and a system for testing a microprocessor that controls, or receives data from other physical apparatus.

...R

Robin2:
I don't agree [that Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause]

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

I want you to walk me through this in case I'm missing something painfully obvious.

Let's say that you have Arduino code, and it reads a digital value from a pin. What possible values can be returned here? My understanding is either a HIGH, or a LOW -- there are no other possibilities.

So, to properly test your code, you will need two test cases: one for when the value of HIGH is returned, and one for when the value of LOW is returned. What happens in the physical world is irrelevant, because you've exhaustively covered the possibilities.

The situation where you're referring to sounds like one where someone makes the assumption that only HIGH or only LOW would always be returned as expected, and only writes one test. That is not a limitation of the arduino_ci library, it is a failure of the test writer. Nothing stops them from enumerating all the combinations that the physical world might throw at them.

I challenge you to post a piece of code that I can't properly unit test. I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.