Example: Unit testing with Catch2

Using Arduino Project Guidance
#1

We run an Escape Room which is, at some places, using Arduinos or bare ATtiny microcontrollers where they control some actuators (like motors), but also effects e.g. on WS2812b LED strips. We need the code to be reliable, and development to be fast – i.e. logic bugs should preferredly be found before downloading the code to the Arduino. So, we unit test the code. It took a few hours to get it working nicely and without polluting the VCS. If you also want to unit test your code, here you go.

Unit Testing?

Unit testing means testing very small code units, normally functions. For example, a function calculates a brightness value based on a timestamp, and the brightness is pulsating at a frequency of 1 Hz. A unit test would verify if it calculates the correct brightness value for the timestamps 0 ms, 100 ms, 600 ms, 1000 ms, 2000 ms (should be the same as 1000 ms and 0 ms). If those values are correct, the probability is high that the function is working correctly.

Unit Testing Arduino Code?

Now how are the IOs and all other Arduino specific functions like sleep() etc. going to be simulated?

The answer is: not. Even though there are libraries which simulate an Arduino to a certain extent, it is easier to limit the scope of the unit tests and exclude the hardware. To state more clearly: Code can be separated into two categories.

  • Hardware Code which interfaces with IOs and interrupts and so on
  • Domain Logic which describes what your program does
    In the previous example, the brightness function is domain logic which calculates the nice pulsating brightness values. The hardware code is then using this brightness function to send the brightness values to the LED aka. hardware.

This differenciation is crucial for unit testing and also enforces writing cleaner code.

Unit Testing with Catch2

Many C++ unit testing frameworks exist. I have chosen Catch2 (reasons are out of scope). The project can easily be copy/pasted for new Arduino projects.

Project on GitHub: GitHub - Granjow/arduino-base: Project structure for Arduino with Catch2 unit testing support

The workflow to get Unit Testing support is as follows:

  • Domain logic code goes to separate files, ideally with header files. For example, demo.cpp and demo.h.
  • The unit tests go to the spec/ directory. Check the Catch2 documentation and the demo.spec.cpp file.
  • For clarity, there is one spec file (unit test file) per domain logic file. Each spec file includes the corresponding domain logic file.
    Domain logic code files that live outside of an Arduino project directory are linked into the project directory (ln -s) as otherwise the compiler will not find them. Keeping domain logic code outside of project directories (and just placing a link there) allows to easier re-use them in different projects. For example, you might have a library for communication between two Arduinos, a sender and a receiver. Both use the same protocol (same .cpp files), but they are two different projects. Using symbolic links avoids code duplication.

Summary

If you want to get started with unit testing and want to give Catch2 a try, here you go! It requires some knowledge about unit testing and using C++ header files, but it definitely improves code quality.

#2

Great topic, this is quite amazing. But I was wondering how I can make GNU to properly compile and understand Arduino firmware.

I’m testing a few things here and I didn’t tested the obvious yet (Get the Arduino.h and #include it in my code and just do a simple g++ -std=c++11 ) but maybe this will not be enough. I was thinking maybe GNU won’t compile it, only AVR-GNU would.

The reason why I’m asking is:
Sometimes when I’m writing functions for Arduino framework I normally use elements of Arduino framework such as String.

and I wanna be able to test those as well.

#3

I don’t actually have any experience with unit testing of Arduino code, other than having run some unit tests and set up the CI system so run them and report code coverage. However, Arduino is also using Catch2 for unit testing of some of their libraries. You can see an example here:
https://github.com/arduino-libraries/ArduinoIoTCloud/tree/master/extras/test
And another:
https://github.com/arduino-libraries/ArduinoCloudThing/tree/master/test

I see this:
https://github.com/arduino-libraries/ArduinoIoTCloud/blob/master/extras/test/include/Arduino.h

#include <string>

/******************************************************************************
   TYPEDEF
 ******************************************************************************/

typedef std::string String;

which is interesting to me, since the API of std::string is not very similar to Arduino’s String class (it’s based on Java’s String).

Regarding the issue of hardware stuff, you can sometimes mock it. Arduino is using FakeIt for this purpose:
https://github.com/eranpeer/FakeIt

#4

@pert This is amazing! I wish I could truly understand half of it. But that's not the case yet.

I will play a bit with their code and try to learn as much as possible, from where I can already tell, they are using Cmake.

I gotta learn Cmake and see if I can end up compiling a unit test code with Catch2, PlatformIO also give a unit test option. but I think the best approach would be to lean how to run Catch2.

#5

You're welcome.

One thing that might be of assistance to you is the commands that run the unit tests during the continuous integration tests of the library (this is the only part of the ArduinoIoTCloud unit test system I set up):

It's a little bit confusing because there are a few extra things going on there beyond just running the unit tests (using valgrind to check for memory leaks, lcov to check the code coverage, and codecov to report and track coverage), but those extra things are very useful once you get the basic unit testing system set up.

To just run the unit tests alone, the only commands required are (run them from the library's root folder):

mkdir extras/test/build
cd extras/test/build
cmake ..
make
bin/testArduinoIoTCloud

I see someone asked for clarification about Arduino's unit test system here:

The person who wrote the tests is away for a bit, so don't hope for an answer there too soon, but definitely worth checking back on it to get the info straight from the source.

As for this:

Or maybe I'm thinking about this the wrong way, and I should be using avr-g++ to compile catch2?

These specific unit tests are run using standard GCC. In fact, ArduinoIoTCloud isn't even compatible with the AVR architecture, so avr-gcc definitely wouldn't be appropriate (the SAMD architecture uses arm-none-eabi-gcc and the ESP8266 architecture uses xtensa-lx106-elf-gcc). So this further shows the value of making the unit tests architecture-independent.

#6