unit testing arduino in eclipse

Before retiring, I successfully introduced unit testing in Eclipse for an embedded arm cross compiled target to my team, based on the book Test-Driven Development for Embedded C by James W. Grenning

Now I'm wanting to do the same for my own Arduino projects, but I've hit a bit of a snag

I have the Eclipse Arduino plugin working on a sample project (the target project) - the functions & classes to be unit tested are in a separate folder in this project (called 'application')

I have the CppUTest unit test libraries installed in a separate testing project
I've included the 'application' folder from the target project in this testing project

Now, This testing project can't immediately resolve the <arduino.h> include file in the target files, and if I manually include this header file into the testing project, it references other include files, and I start to disappear down a rabbit hole trying to satisfy other header file references

I've initially worked around this by removing <arduino.h> from the target functions and classes, and the only changes needed so far has been to change Arduino type defs like 'byte' to standard C 'unsigned char' - these then copile fine in both projects ad the tests run and pass

But I don't particularly like having to change the type defs, and would like to be able to include <arduino.h>, and keep the code in the usual Arduino 'style'

So, the question is - is there a better way to do this?

I did think that I should be able to add the CppUTest libraries to an Arduino Eclipse project, as a test project, but there doesn't seem to be the option to add library & include paths to an Arduino project like you can with a c/c++ project, and I don't see an easy way to add all the necessary Arduino header files to a standard C++ project, especially as I believe some of the headers are dependant on the Arduino platform chosen, which of course isn't supported in a standard C++ project

It would be nice to have a generic solution to this so that I could create other test projects with different unit testing frameworks, as my former colleagues have moved onto the Cxx unit test framework, and there are others (like google) i'd like to try

Looking forward to your suggestions

Let me start by saying i am no expert. But i like to give my 2 cents in the hope it helps.

Would there be a problem if u make your own Arduino.h and use that one to define the types you need?

Are u willing to share your testing setup / example code? I have been playing a bit with it but never got it running. Do you also have emulation for IO/Serial stuff or only logic?

Hi Pimmetje

Straight after posting the question I did make my own Arduino.h, and added it to the include path in the project settings so the #include<> directive found it
I guess adding new elements to this 'mock' Arduino.h whenever I use new bits is not too bad - just not as 'clean' as I'd like

I'm only testing MY logic, I'm not trying to test inbuilt functions like digitalWrite
One of my aims in unit testing is to encourage the splitting of functionality into functions/classes with single responsibility - which are then much easier to create unit tests for

Hence the call to turn the inbuilt LED on/off is

digitalWrite(LED_BUILTIN, sysLED.GetLEDStatus(LED_timer)); //update system LED

And is in the main .ino file, which isn't tested (and doesn't contain any of my logic, just inbuilt function calls to the hardware)

But the class sysLED which has the function GetLEDStatus is in the SystemLED class file which is tested

Don't have a problem sharing my setup - what exactly would you like? a zip of everything?

I had to compile CppUTest on my machine from the github source - so the library file might not work for you depending on the setup of eclipse/GCC you have, but happy to share

Further to testing I/O - in my previous life we setup a type of functional or black box testing usually called 'hardware in the loop', where we hooked up the target hardware to a pc using programmable I/O modules, and automatically uploaded new firmware onto the target from the build server, and then ran tests that examined the outputs of the target depending on the inputs forced on it by the test program running on the PC - including varying the supply voltage to the target to conform it worked within stated limits
Its an expensive way of testing and takes longer to produces the tests (and run them), but essential when you're aiming to roll out 'over the air' software updates to hundreds of thousands of remote units, where a fault that could 'brick' all the units would be terminal for the business - when I retired we where running over 10,000 tests for each new firmware release
I think this would be massive overkill for typical Arduino projects, as its more suited to projects that are constantly updated with new features, as we where doing

I would like to know what version of the CPP Testframe u used.
The commands for compiling / running would do in my case.

There are some other people who made mockups (selected some randomly).

But i guess running the real thing is the only way to proper test a device. But that involves a lot of work that i would rather spent on the hobby project itself. But sadly there are no emulator that allow running a hexfile.

Thanks for your reply.

Another question (if i may).

How are u versioning your libraries and do you have a auto-build/CI server? I currently run a gitlab-ci with multiple projects sharing one big library folder.
Some classes are in the project scope and if useful the logic classes are in the global library.

Pimmetje:
I would like to know what version of the CPP Testframe u used.
The commands for compiling / running would do in my case.

I've noted below what I remember that I did to set this up - let me know if you want more detail on any specific item
I've probably missed something out, so apologies in advance if I have

Install Eclipse CDT 2019-06
Install MinGW GCC - Setting up MinGW-w64

get CppUTest & build to the attached instructions

Add Eclipse C++ ide for Arduino 3.0 - From eclipse market place

Make target Arduino project
Create sub folder for files to be tested

Make C++ test project
Add Cpputest libraries, library path & includes to test project
Add target sub folder from target project to test project
Create tests (look in the example folder of CppUTest)

Pimmetje:
Another question (if i may).

How are u versioning your libraries and do you have a auto-build/CI server? I currently run a gitlab-ci with multiple projects sharing one big library folder.
Some classes are in the project scope and if useful the logic classes are in the global library.

I've not gotten around to building libraries in my home Arduino projects yet, but at work we had someone (who knew more about subversion that I do) add something that updated one of the files with the SVN version so it was baked into the build & added to the output filename- I particularly like this because at any point in the future you can checkout and inspect or rebuild that specific version from SVN
Not sure how you do that with GIT, but I'm sure it's possible
On some projects we also had manually updated semantic versioning, but I liked to always have the SVN version compiled in as someone will eventually forget to update the semantic versioning, and it never seems easy to identify a specific check-in from an unrelated semantic version number
The major advantage with semantic versioning in my view is that you can use it for programmatic compatibility checking

I don't have a build server at home, but again at work I originally setup CruiseControl as the build server, but one of my guys migrated it all to Jenkins and added more post build steps for all the different test stages - Jenkins did seem more reliable and easier to configure

Hope that's what you were looking for

Pimmetje:
But i guess running the real thing is the only way to proper test a device. But that involves a lot of work that i would rather spent on the hobby project itself. But sadly there are no emulator that allow running a hexfile.

Yes, I think you have to be very pragmatic as to why you're adding tests - its very easy to add tests that have no practical value or aren't worth the effort in adding them
I personally think unit tests are particularly useful to test functions work as expected with all possible input values, as it allows you to explore what happens with values not possible in the real world - so unexpected values don't cause unexpected behaviour

I don't know. for Arduino sketch? You use professional tools and project setup for Arduino? Why don't you use professional frameworks and SDK for MCU software development?

@David128 Thanks for your input, ill have a try on that.
As for the libraries & builds. I have some special strings in my project that my CI server will replace. So all my projects will print there project name & build number out of the serial port. This is very handy if u want to update a project that is old. This way u can identify the version before replacing it and roll back if needed.
It also has the version hash of the library project in my case. So i can always rebuild the exact same build if i need to.

@juraj I personally like arduino because it's simple to work with and to setup. I can make things very complex and still be able to work with them. But in my case other people have to be able to understand it. None of them are professional programmers. But things like CI i can do how i want because it's optional and server side.

Juraj:
I don't know. for Arduino sketch? You use professional tools and project setup for Arduino? Why don't you use professional frameworks and SDK for MCU software development?

I understood that Arduino grew out of some form of teaching project, and I strongly believe that we ought to be exposing students to 'professional' programming methods - and my personal recommendation would be Test Driven Development
In my opinion, unit testing doesn't just test the code, but strongly encourages better design of software, which people learning to code ought to understand

@David thanks for pointing me in the right direction. (Sorry for the late reply). I did some test with it. Unittests are nice :). I hit a limit within minutes (i wanted to test a parser). But to save memory it uses the F() macro. Boom. Unittest does not work because it's avr special. Now i am going to try to get the GitHub - ianfixes/arduino_ci: Unit testing and Continuous Integration (CI) for Arduino libraries, from a Ruby gem to work. Maybe it can do it. Maybe not. But it has some nice features i am going to try :).