Running automated unit tests on an Arduino library using Travis CI

ifreecarve:
And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously

It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

Does the Atmel development system support unit testing?

...R

I'm glad you're excited about it. Work on Arduino hardware packages was what got me initially interested in testing automation because it was just such a huge amount of work just to compile all the possible combinations manually that the job of testing was sometimes ending up getting put on the users. That's fine with willing beta testers but it's a bad experience for unsuspecting beginners who probably assume the bug they ran across is something they are doing wrong.

Generally a hardware package repository will be structured so that it can just be directly copied into the hardware subfolder of the Arduino sketchbook folder for manual installation. The folder structure required for Boards Manager installation is a little bit different in that there is no architecture folder so everything is moved up one folder level. Some repositories do have the Boards Manager installation structure, for example: ESP8266, ESP-32, nRF-52. All three of those require toolchain installation so they would need to go through some extra package installation steps anyway and thus structuring the repo for easy manual installation is not really beneficial.

Robin2:
It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

I do agree that the target Arduino user would only be overwhelmed by the concept of unit testing. However, I think it is reasonably in reach for the average 3rd party Arduino library or hardware package developer, especially if people like ifreecarve help to make it more accessible. I've been noticing compilation tests using services like Travis CI becoming more common in the Arduino world but unit tests are pretty rare. I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss. Being able to check the results of the Travis CI build is a great way for a potential user to do a quick initial evaluation.

I agree that unit testing an Arduino sketch is beyond the market of hobbyists that Arduino is aiming at.

On the other hand,a brief sampling of Arduino projects turns up smoke/fire alarms, automotive projects, heavy equipment control, hydraulics, drones, healthcare, etc. Software errors on projects like these can cause real harm. Obligatory Therac-25 reference goes here.

I don't have a problem with a lack of ability to test individual sketches; each person can take responsibility for their own work. However, the practice of using libraries -- code provided by complete strangers via the Internet -- without an ability to evaluate their quality or correctness, does cross a line into being a Bad Idea for some applications.

We should at least have the option to create tests for library code, regardless of whether the majority of library writers will make use of it.

pert:
I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss.

I have not come across libraries that simply don't work. My experience with libraries and from reading about Forum members using libraries is that there are two main problems

  • Totally inadequate documentation
  • Incompatibility between libraries.

I don't believe unit testing can address either of those problems.

To some extent it may never be possible to eliminate all incompatibilities between libraries simply because there is a very limited set of hardware that they must all share.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

...R

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

I think that's the case for this project, but if you try it out and believe otherwise then I would welcome your feedback.

Robin2:
I have not come across libraries that simply don't work.

I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

Robin2:
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

Agreed. The GUI that the average user interacts with must be kept very simple. But the Arduino IDE can allow for advanced usage under the hood. Regarding the test folder thing (which is the only specific change that's been mentioned) I'm not sure it would even end up as a change to the current IDE code but only a change in the text of the library specification. Currently the Arduino IDE has a source folder whitelist approach, meaning that there shouldn't be any problem putting the unit tests in the test folder. I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

pert:
I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

Yes, that's accurate. In Java, there would be a "src" directory for the code and "test" directory for the tests. In Python the code would live in a directory that's the same as the library name, and the tests in "test". Ruby would have a "lib" and a "spec" directory, and JavaScript a "src" and "test" directory. In Swift, code would be split among a number of directories but tests live in "Tests". Etc.

I don't think it makes any sense to clutter the Arduino IDE. Adding a whitelist entry for "test" to the library specification would be best-case-scenario for me, including (and/or especially) if they didn't prescribe a particular test tool to manage that. It would be up to the folks who want to do collaborative development via GitHub to implement their own tests and CI in that directory.

pert:
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

I suspect part of the problem is that the authors of libraries don't have access to a wide range of Arduino boards.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?

I think I would be content with decent documentation that, inter alia, lists the boards that the author has tested the library on. And I don't mean unit tests.

If people can't be bothered to write documentation what chance is there that they will take the trouble to learn how to do unit testing.

...R

Well I think one reason developers don't like writing documentation is because, even though they're sharing it, the code is primarily written for themselves and they think (albeit wrongly) of documentation as being an extra task solely for the benefit of other users. However, as a user of the code, there is an obvious direct benefit of having it be thoroughly and efficiently tested.

Automated testing also makes writing code much more enjoyable because you get to do just the fun, creative part that humans are best at and leave all the boring stuff that machines are best at to the computer.

The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way. In fact that's how I've been with unit testing. I've been thinking about it a lot lately but there hasn't been a clear path to actually getting going with it. Maybe arduino_ci will help me finally get there.

I've been working on making it easier to do continuous integration compilation tests of Arduino projects with the hopes that if that initial barrier is lowered then more people will start doing it. So far I don't think I've made any difference but I'll keep working at it.

pert:
The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way.

I don't reckon I do enough Arduino programming to get the pay-off that you describe.

Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

So much of a microprocessor program depends on the physical world outside which cannot be tested using software. And building another microprocessor to simulate the real world is fraught with difficulties, even if there was time to do so.

You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.

In theory I can see a lot of value in test-driven-development - building the test before you write the code. In practice I am lazy.

I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.

...R

Robin2:
Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

Yes, my understanding is that it is a good bit of work writing unit tests. It shouldn't be too bad if you write them as you go along but for existing projects it's going to take a real effort. Even so, it's not all or nothing. You can start with the highest priority tests and add coverage gradually over time.

The compilation tests are a much lower hanging fruit. In my case it mostly consists of having Travis CI use the Arduino IDE to compile library examples. If the examples compile then the build passes. If one of them doesn't compile then the build fails. The trouble is that doesn't tell you whether the code actually works, only that it compiles. I found there were still some tricky parts to doing those compilation tests with Travis CI in the way I wanted so I ended up wrapping all that up in a Bash script, which makes it even easier to set up that sort of continuous integration. It saves a report of the results of the compilations for each build, which makes it easy to compare changes between commits, such as checking if the flash or global SRAM memory usage changed. I can also compile with multiple versions of the Arduino IDE for multiple boards to make sure no incompatibilities were introduced.

Robin2:
I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

Sure, but then afterwards you can write a test for it so it will never bite you again.

Robin2:
You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.

Yes, that's the whole idea behind arduino_ci. I'm still trying to wrap my head around it but you can set and read the states of virtual pins.

Robin2:
In practice I am lazy.

Me too, and what a great feeling it is to kick back and watch Travis CI running thousands of compilations for me.

Robin2:
I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.

Yeah, it's especially nice for open source projects where you get random pull requests. The CI tests automatically run on the pull request so you have this initial triage before you even need to take a look at the proposal. If it doesn't build then the person who submitted the PR can look at the build, find the bug, and then update the PR until it passes the build. A repository maintainer still will need to look at it before it can be merged but that initial bug shakedown can save them a lot of time and effort.

pert:
Yeah, it's especially nice for open source projects where you get random pull requests

That's way way beyond my pay grade :slight_smile:

...R

Robin2:
Would it be possible for an author to run tests that verify compatibility with a board that he does not have?

Yes, arduino_ci does this.

In fact, I was able to develop this system without connecting a single piece of Arduino hardware to my laptop. "Compatibility with a board" is nothing more than setting the right set of preprocessor defines for the compiler. This in turn causes various features to be enabled or disabled (e.g. serial ports).

You can validate this for yourself by attempting to run serial comms unit tests on a board that doesn't have a serial port.

Robin2:
So much of a microprocessor program depends on the physical world outside which cannot be tested using software.

Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause. I provide this power, in a feature that I've termed "GODMODE". If there's a physical scenario that GODMODE can't replicate, then I'll be glad to update the library appropriately.

ifreecarve:
Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.

I don't agree with that at all.

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

To be honest, I don't believe using another microprocessor to simulate the external world would be sufficient.

To my mind there is a huge difference between a test system for (say) a program running on a web server where all the input consists of messages from another computer program and a system for testing a microprocessor that controls, or receives data from other physical apparatus.

...R

Robin2:
I don't agree [that Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause]

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

I want you to walk me through this in case I'm missing something painfully obvious.

Let's say that you have Arduino code, and it reads a digital value from a pin. What possible values can be returned here? My understanding is either a HIGH, or a LOW -- there are no other possibilities.

So, to properly test your code, you will need two test cases: one for when the value of HIGH is returned, and one for when the value of LOW is returned. What happens in the physical world is irrelevant, because you've exhaustively covered the possibilities.

The situation where you're referring to sounds like one where someone makes the assumption that only HIGH or only LOW would always be returned as expected, and only writes one test. That is not a limitation of the arduino_ci library, it is a failure of the test writer. Nothing stops them from enumerating all the combinations that the physical world might throw at them.

I challenge you to post a piece of code that I can't properly unit test. I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.

ifreecarve:
I challenge you to post a piece of code that I can't properly unit test. I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.

Sorry, I know it is lazy of me, but I am not sufficiently interested to go to that much trouble.

Maybe you are right in that the problem lies with the person writing the tests rather than the test software itself.

A common problem that arises in the Forum is switch bounce - I have to say I can't think how I would write a test to prove that my program could deal with that.

Also, regarding switch bounce, it is not always necessary to deal with it. And in some cases the problem does not arise and does not need to be dealt with - and dealing with it when it is not needed just complicates a program and wastes CPU cycles.

I reckon you need a test which first of all determines if there is a switch bounce problem.

A program may work perfectly well with one brand of switch and fail miserably with a different brand.

...R

Robin2:
I reckon you need a test which first of all determines if there is a switch bounce problem.

You've phrased this in a strange way. I'd imagine that every program has a theoretical "minimum acceptable delay between state changes", and whether or not they guard against that (e.g. handling input from a bouncing switch) is entirely testable. If you purposefully choose not to implement those guards for the purposes of reducing complexity or CPU cycles, then that's your answer -- no test required.

Your concerns about my work are quite valid by the way, and I welcome them. After all, I'm making some bold claims about what I can test, which run counter to a lot of prevailing wisdom on what is and isn't possible/practical in an Arduino system. However, without a specific example that shows the problem you describe (i.e. substantiating your claims), your previous comments on that topic just come off as bluster.

ifreecarve:
your previous comments on that topic just come off as bluster.

I can understand that.

I will remain with the doubters.

...R

Robin2:
A common problem that arises in the Forum is switch bounce - I have to say I can't think how I would write a test to prove that my program could deal with that.

It's really not difficult.

Consider the example debouncing code provided by Arduino's website; I've delegated that code to a function called onLoop(), as you might find in a library implementation.

[size=0.8em]Code: [url=https://arduinogetstarted.com/tools/arduino-code-highlighter]see how to post code[/url] [/size]

---



```
[size=0.8em][nobbc]// pretend that the "debounce" sketch is a library function that is run on every loop[/nobbc]
[nobbc]// e.g.[/nobbc]
[nobbc]//  void loop() { onLoop(); }[/nobbc]
[nobbc]//[/nobbc]
const int[nobbc] buttonPin = 2;          [/nobbc][nobbc]// the number of the pushbutton pin[/nobbc]
const int[nobbc] ledPin = 13;            [/nobbc][nobbc]// the number of the LED pin[/nobbc]
const long[nobbc] debounceDelay = 50;    [/nobbc][nobbc]// debounce time; increase if the output flickers[/nobbc]
int[nobbc] ledState;                    [/nobbc][nobbc]// current state of the output pin[/nobbc]
int[nobbc] buttonState;                  [/nobbc][nobbc]// current reading from the input pin[/nobbc]
int[nobbc] lastButtonState;              [/nobbc][nobbc]// previous reading from the input pin[/nobbc]
unsigned long[nobbc] lastDebounceTime;  [/nobbc][nobbc]// last time the output pin was toggled[/nobbc]

void[nobbc] onLoop() {[/nobbc]
  [nobbc]// read state, record time if the input flipped[/nobbc]
  int[nobbc] reading = [/nobbc]digitalReadnobbc;[/nobbc]
  if[nobbc] (reading != lastButtonState) lastDebounceTime = [/nobbc]millisnobbc;[/nobbc]

if[nobbc] (([/nobbc]millisnobbc - lastDebounceTime) > debounceDelay) {[/nobbc]
    if[nobbc] (reading != buttonState) {[/nobbc]
      buttonState = reading;
      if[nobbc] (buttonState == [/nobbc]HIGH[nobbc]) ledState = !ledState;[/nobbc]
      digitalWrite[nobbc](ledPin, ledState);[/nobbc]
    }
  }

lastButtonState = reading;
}[/size]
```

|

Here is one unit test that corresponding to one possible bounce case: a single bounce on the button pin (starts low, bounces from high to low, settles on high):

[size=0.8em]Code: [url=https://arduinogetstarted.com/tools/arduino-code-highlighter]see how to post code[/url] [/size]

---



```
[size=0.8em]unittest(bounce_low_to_high) {
[nobbc]  ledState                    = [/nobbc]LOW;
[nobbc]  buttonState                  = [/nobbc]LOW;
[nobbc]  lastButtonState              = [/nobbc]LOW;
lastDebounceTime            = 0;

[nobbc]  assertEqual(1, state->digitalPin[ledPin].historySize()); [/nobbc][nobbc]// pin history contains only the initial LOW state[/nobbc]

[nobbc]  state->[/nobbc]micros[nobbc]                = 25000;[/nobbc]
[nobbc]  state->digitalPin[buttonPin] = [/nobbc]HIGH[nobbc];                    [/nobbc][nobbc]// set initial button entry to HIGH[/nobbc]
  onLoop();
[nobbc]  assertEqual(25, lastDebounceTime);                      [/nobbc][nobbc]// debounce time has reset[/nobbc]
[nobbc]  assertEqual([/nobbc]LOW[nobbc], state->digitalPin[ledPin]);            [/nobbc][nobbc]// no change in output[/nobbc]
[nobbc]  assertEqual([/nobbc]HIGH[nobbc], lastButtonState);[/nobbc]

[nobbc]  state->[/nobbc]micros[nobbc]                = 50000;[/nobbc]
[nobbc]  state->digitalPin[buttonPin] = [/nobbc]LOW[nobbc];                      [/nobbc][nobbc]// bounce button LOW[/nobbc]
  onLoop();
[nobbc]  assertEqual(50, lastDebounceTime);                      [/nobbc][nobbc]// debounce time has reset[/nobbc]
[nobbc]  assertEqual([/nobbc]LOW[nobbc], state->digitalPin[ledPin]);            [/nobbc][nobbc]// no change in LED output[/nobbc]
[nobbc]  assertEqual([/nobbc]LOW[nobbc], lastButtonState);[/nobbc]

[nobbc]  state->[/nobbc]micros[nobbc]                = 75000;[/nobbc]
[nobbc]  state->digitalPin[buttonPin] = [/nobbc]HIGH[nobbc];                    [/nobbc][nobbc]// bounce button HIGH[/nobbc]
  onLoop();
[nobbc]  assertEqual(75, lastDebounceTime);                      [/nobbc][nobbc]// debounce time is again reset[/nobbc]
[nobbc]  assertEqual([/nobbc]LOW[nobbc], state->digitalPin[ledPin]);            [/nobbc][nobbc]// still no change in LED output[/nobbc]
[nobbc]  assertEqual([/nobbc]HIGH[nobbc], lastButtonState);[/nobbc]

[nobbc]  state->[/nobbc]micros[nobbc]                = 126000;                  [/nobbc][nobbc]// actual boundary case, time elapsed[/nobbc]
[nobbc]  state->digitalPin[buttonPin] = [/nobbc]HIGH;
  onLoop();
[nobbc]  assertEqual(75, lastDebounceTime);                      [/nobbc][nobbc]// no additional bounce happened[/nobbc]
[nobbc]  assertEqual([/nobbc]HIGH[nobbc], state->digitalPin[ledPin]);            [/nobbc][nobbc]// therefore the LED turns on[/nobbc]
[nobbc]  assertEqual(2, state->digitalPin[ledPin].historySize()); [/nobbc][nobbc]// digital output was written only once[/nobbc]
}[/size]
```

|

There is absolute control over "when" the pins are read and what values they have, so you are free to implement however many use cases you can conceive of.

The entire working example (the function under test followed by its unit tests) that I wrote can be found at arduino_ci/SampleProjects/TestSomething/test/debounce.cpp at master · Arduino-CI/arduino_ci · GitHub

An example of the CI running it can be found at https://github.com/Arduino-CI/arduino_ci/runs/1650531936?check_suite_focus=true#step:4:573

Does this answer your question of proof? What bounce behaviors would you want to see simulated, or which ones don't you believe to be possible?