Arduino Forum

Development => Other Software Development => Topic started by: ifreecarve on Mar 07, 2018, 11:22 pm

Title: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 07, 2018, 11:22 pm
I'm pleased to announce a system for unit testing Arduino libraries (https://github.com/ifreecarve/arduino_ci) capable of supporting continuous integration (CI) on GitHub / Travis CI.  The purpose of this is to make it easier to accept code contributions to your Arduino library on GitHub from complete strangers -- you don't have to bother downloading / compiling / uploading their code in order to test it manually.

Instead, you let the computer do the testing for you.

In addition to the writeup I did on StackOverflow (https://stackoverflow.com/a/49161903/2063546),  the project README (https://github.com/ifreecarve/arduino_ci/blob/master/README.md) is fairly up to date and I have a few working examples (Arduino-Queue (https://github.com/sdesalas/Arduino-Queue.h/pull/1/files), Adafruit FONA (https://github.com/adafruit/Adafruit_FONA/pull/91/files)) to show it in action.

Is this sort of project relevant for this community, or are things like GitHub collaboration and Continuous Integration beyond the scope of practical Arduino work?
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 07, 2018, 11:58 pm
More thorough and easier testing of Arduino projects will lead to less bugs, the developers being able to focus on the authors being able to make improvements rather than spending their time on testing and addressing bug reports, and will generally make things easier for all the people providing support to Arduino users. For these reasons I think your project is a great contribution to the Arduino community. Thanks!

I've been spending a lot of time getting familiar with the use of Travis CI on Arduino projects and have been trying to help add this capability to some other Arduino projects. I have not yet started my journey into unit tests but that is definitely on my to-do list after I've reached a more finished stage in my current CI projects. So I haven't yet used your project but I do have a couple of comments based solely on my reading of your documentation. Forgive me if I have missed something obvious.

I notice that your system expects the unit tests to be located under the test subfolder of the library. The Arduino Library Specification states:
https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#extra-documentation (https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification#extra-documentation)
Quote
An extras folder can be used by the developer to put documentation or other items to be bundled with the library.
...
The content of the extras folder is totally ignored by the IDE; you are free to put anything inside such as supporting documentation, etc.
So it seems like extras/test would be the recommended location for internal testing files to be placed.

One place where automated testing is especially useful is for Arduino hardware packages. Typically these will include architecture specific libraries and testing these libraries against the various board configurations of the package can be extremely time consuming if done manually. If your project supports it, I think it would be helpful to add a demonstration of that usage.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 08, 2018, 02:43 pm
I'm not sure how best to handle the "extras" folder.  On one hand, it would be more consistent with the documentation as you pointed out.  On the other hand, it would make the definitions of the tests a bit more kludgy by placing them further from the code that they're testing.  And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously and just have the IDE consider the "test" folder to be a normal and welcome addition to a project.

I don't expect that to happen in the foreseeable future though.

Can you give some examples of what you mean by a hardware package?  I do have some capabilities to define compiler flags and custom platforms to test on, but I'm not sure whether that relates to what you're talking about.  Do you have a link to the library in question?
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 08, 2018, 02:59 pm
Here are some popular examples of what I call "hardware packages" (also known as "cores"):
https://github.com/MCUdude/MiniCore (https://github.com/MCUdude/MiniCore)
https://github.com/MCUdude/MicroCore (https://github.com/MCUdude/MicroCore)
https://github.com/MCUdude/MegaCore (https://github.com/MCUdude/MegaCore)
https://github.com/MCUdude/MightyCore (https://github.com/MCUdude/MightyCore)
https://github.com/MCUdude/MajorCore (https://github.com/MCUdude/MajorCore)
https://github.com/SpenceKonde/ATTinyCore (https://github.com/SpenceKonde/ATTinyCore)
https://github.com/sandeepmistry/arduino-nRF5 (https://github.com/sandeepmistry/arduino-nRF5)
https://github.com/espressif/arduino-esp32 (https://github.com/espressif/arduino-esp32)
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 08, 2018, 04:36 pm
Hmm, if I'm reading this right then you're talking about running hardware tests on the board itself.  I don't think I can do that -- I strip out all the hardware-related functionality and provide software substitutes for it.  Only the board constants get pulled in (#defines for comm port pins, those sort of things).
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 08, 2018, 04:41 pm
I'm not talking about running tests on the physical hardware. I'm talking about doing unit tests of the libraries bundled with the hardware packages. For example:
https://github.com/MCUdude/MightyCore/tree/master/avr/libraries (https://github.com/MCUdude/MightyCore/tree/master/avr/libraries)
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 08, 2018, 04:45 pm
Aah!  Yes, this should be possible.  But possible-ish... I'm not sure how I'd get the cores integrated with the codebase.   I had to pull in a lot of the Arduino header files manually to get things to compile properly:

https://github.com/ifreecarve/arduino_ci/tree/master/cpp/arduino/avr (https://github.com/ifreecarve/arduino_ci/tree/master/cpp/arduino/avr)

It's not immediately clear how I'd get a custom set of headers to integrate with that.  When you develop such a library, where do you put it?  Is it something that could be downloaded as a package from a custom source?   I have the ability to do that:

https://github.com/ifreecarve/arduino_ci/blob/master/misc/default.yml#L5-L10 (https://github.com/ifreecarve/arduino_ci/blob/master/misc/default.yml#L5-L10)
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 08, 2018, 04:53 pm
Well the goal would be to install the hardware package as it exists in the current development state. Some hardware packages do have the JSON file for Boards Manager installation but that will install a release version of the package rather than the tip. So, for example, for the compilation tests of the MightyCore bundled library examples sketches via Travis CI the first step is to install the Arduino IDE and then copy the contents of the repository to the hardware subfolder of the sketchbook folder. This is known as the manual installation method, as opposed to the Boards Manager installation method. Then the compilations are done not only with the latest versions of the library code but also the latest version of the hardware package.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 08, 2018, 05:20 pm
I think you sort of answered your own question, you'd need to either configure things locally such that the yml file described what was on your disk, OR, automate the steps to do that as part of the Travis CI build script.

The good news is that my script only installs the Arduino IDE if it can't find it.  https://github.com/ifreecarve/arduino_ci/blob/master/lib/arduino_ci/arduino_installation.rb#L87-L96 (https://github.com/ifreecarve/arduino_ci/blob/master/lib/arduino_ci/arduino_installation.rb#L87-L96)

So this should be workable.  Want to try this out?  
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 09, 2018, 01:07 am
Just to clarify this a bit: my system lets you define platforms, and platforms are based on packages.  This is done in the .arduino-ci.yml file.

If the board/core packages you want are known to your Arduino IDE already (like, they are installed in a local directory that you've already set up), no problem -- define a platform in the yml file that refers to that package.  If not, your options are to either specify the package info in the yml file (the URL for them), or to circumvent that requirement by setting the Travis CI script to install the files you need in the same location that they'd exist on your local machine.

Does this make any sense?  TL;DR I think it will work.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 09, 2018, 03:06 am
Yes, it makes perfect sense. I haven't had a chance to try it out yet but will do so ASAP. Thanks.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 09, 2018, 11:57 am
No rush, I only doubled my reply because I re-read my initial one and realized that it was a very poor answer to your question :)

You'll have to excuse my excitement though; this wasn't a feature that I had in mind when I wrote the system, and it makes the extra effort I put into the yaml file layout seem much more worth it.  As much as I'm interested in whether it works for you, I'm also interested in knowing all the steps you have to take in order to make it work.  

My thinking is that it might be possible for me to add a config entry to the YAML file that would let you specify a path on disk (or some other locator) for a still-in-development hardware package.  Then it's just a question of how to act on that information (e.g. forcibly copying it into the Arduino IDE each time, in case changes were made to it).  I just need to learn more about the development cycle for hardware packages.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 09, 2018, 12:09 pm
And ultimately, I'd love it if the Arduino project itself would take unit testing more seriously
It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.

Does the Atmel development system support unit testing?

...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 09, 2018, 12:38 pm
I'm glad you're excited about it. Work on Arduino hardware packages was what got me initially interested in testing automation because it was just such a huge amount of work just to compile all the possible combinations manually that the job of testing was sometimes ending up getting put on the users. That's fine with willing beta testers but it's a bad experience for unsuspecting beginners who probably assume the bug they ran across is something they are doing wrong.

Generally a hardware package repository will be structured so that it can just be directly copied into the hardware subfolder of the Arduino sketchbook folder for manual installation. The folder structure required for Boards Manager installation is a little bit different in that there is no architecture folder so everything is moved up one folder level. Some repositories do have the Boards Manager installation structure, for example: ESP8266, ESP-32, nRF-52. All three of those require toolchain installation so they would need to go through some extra package installation steps anyway and thus structuring the repo for easy manual installation is not really beneficial.

It strikes me that that is quite a distance beyond the market that the Arduino system is aimed at.
I do agree that the target Arduino user would only be overwhelmed by the concept of unit testing. However, I think it is reasonably in reach for the average 3rd party Arduino library or hardware package developer, especially if people like ifreecarve help to make it more accessible. I've been noticing compilation tests using services like Travis CI becoming more common in the Arduino world but unit tests are pretty rare. I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss. Being able to check the results of the Travis CI build is a great way for a potential user to do a quick initial evaluation.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 09, 2018, 01:29 pm
I agree that unit testing an Arduino sketch is beyond the market of hobbyists that Arduino is aiming at. 

On the other hand,a brief sampling of Arduino projects turns up smoke (https://create.arduino.cc/projecthub/icstation-team/diy-temperature-humidity-smoke-detector-alarm-system-bas-5039f7?ref=search&ref_id=smoke&offset=4)/fire (https://create.arduino.cc/projecthub/samsung-iot/monitor-fire-and-temperature-using-artik-cloud-f1b5ec) alarms, automotive projects (https://www.kickstarter.com/projects/1029808658/macchina-the-ultimate-tool-for-taking-control-of-y), heavy equipment control (http://forum.arduino.cc/index.php?topic=17209.0), hydraulics (https://create.arduino.cc/projecthub/ajartech/the-liberator-compressed-earth-brick-press-ced8b5), drones, healthcare (https://create.arduino.cc/projecthub/Technovation/health-band-a-smart-assistant-for-the-elderly-0fed12?ref=search&ref_id=healthcare&offset=5), etc.  Software errors on projects like these can cause real harm.  Obligatory Therac-25 reference goes here. (https://en.wikipedia.org/wiki/Therac-25)

I don't have a problem with a lack of ability to test individual sketches; each person can take responsibility for their own work.  However, the practice of using libraries (https://www.arduino.cc/en/Guide/Libraries#toc3) -- code provided by complete strangers via the Internet -- without an ability to evaluate their quality or correctness, does cross a line into being a Bad Idea for some applications. 

We should at least have the option to create tests for library code, regardless of whether the majority of library writers will make use of it. 
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 09, 2018, 03:35 pm
I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss.
I have not come across libraries that simply don't work. My experience with libraries and from reading about Forum members using libraries is that there are two main problems

I don't believe unit testing can address either of those problems.

To some extent it may never be possible to eliminate all incompatibilities between libraries simply because there is a very limited set of hardware that they must all share.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 09, 2018, 08:33 pm
Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.
I think that's the case for this project, but if you try it out and believe otherwise then I would welcome your feedback.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 09, 2018, 08:42 pm
I have not come across libraries that simply don't work.
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.
Agreed. The GUI that the average user interacts with must be kept very simple. But the Arduino IDE can allow for advanced usage under the hood. Regarding the test folder thing (which is the only specific change that's been mentioned) I'm not sure it would even end up as a change to the current IDE code but only a change in the text of the library specification. Currently the Arduino IDE has a source folder whitelist approach, meaning that there shouldn't be any problem putting the unit tests in the test folder. I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 09, 2018, 09:17 pm
I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?
Yes, that's accurate.  In Java, there would be a "src" directory for the code and "test" directory for the tests.  In Python the code would live in a directory that's the same as the library name, and the tests in "test".  Ruby would have a "lib" and a "spec" directory, and JavaScript a "src" and "test" directory.   In Swift, code would be split among a number of directories but tests live in "Tests".  Etc.

I don't think it makes any sense to clutter the Arduino IDE.  Adding a whitelist entry for "test" to the library specification would be best-case-scenario for me, including (and/or especially) if they didn't prescribe a particular test tool to manage that.  It would be up to the folks who want to do collaborative development via GitHub to implement their own tests and CI in that directory.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 09, 2018, 11:32 pm
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.
I suspect part of the problem is that the authors of libraries don't have access to a wide range of Arduino boards.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?


I think I would be content with decent documentation that, inter alia, lists the boards that the author has tested the library on. And I don't mean unit tests.

If people can't be bothered to write documentation what chance is there that they will take the trouble to learn how to do unit testing.

...R

Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 10, 2018, 12:02 am
Well I think one reason developers don't like writing documentation is because, even though they're sharing it, the code is primarily written for themselves and they think (albeit wrongly) of documentation as being an extra task solely for the benefit of other users. However, as a user of the code, there is an obvious direct benefit of having it be thoroughly and efficiently tested.

Automated testing also makes writing code much more enjoyable because you get to do just the fun, creative part that humans are best at and leave all the boring stuff that machines are best at to the computer.

The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way. In fact that's how I've been with unit testing. I've been thinking about it a lot lately but there hasn't been a clear path to actually getting going with it. Maybe arduino_ci will help me finally get there.

I've been working  (https://github.com/per1234/arduino-ci-script)on making it easier to do continuous integration compilation tests of Arduino projects with the hopes that if that initial barrier is lowered then more people will start doing it. So far I don't think I've made any difference but I'll keep working at it.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 10, 2018, 10:51 am
The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way.
I don't reckon I do enough Arduino programming to get the pay-off that you describe.

Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

So much of a microprocessor program depends on the physical world outside which cannot be tested using software. And building another microprocessor to simulate the real world is fraught with difficulties, even if there was time to do so.


You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.


In theory I can see a lot of value in test-driven-development  -  building the test before you write the code. In practice I am lazy.


I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.


...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: pert on Mar 10, 2018, 11:45 am
Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.
Yes, my understanding is that it is a good bit of work writing unit tests. It shouldn't be too bad if you write them as you go along but for existing projects it's going to take a real effort. Even so, it's not all or nothing. You can start with the highest priority tests and add coverage gradually over time.

The compilation tests are a much lower hanging fruit. In my case it mostly consists of having Travis CI use the Arduino IDE to compile library examples. If the examples compile then the build passes. If one of them doesn't compile then the build fails. The trouble is that doesn't tell you whether the code actually works, only that it compiles. I found there were still some tricky parts to doing those compilation tests with Travis CI in the way I wanted so I ended up wrapping all that up in a Bash script, which makes it even easier to set up that sort of continuous integration. It saves a report of the results of the compilations for each build, which makes it easy to compare changes between commits, such as checking if the flash or global SRAM memory usage changed. I can also compile with multiple versions of the Arduino IDE for multiple boards to make sure no incompatibilities were introduced.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.
Sure, but then afterwards you can write a test for it so it will never bite you again.

You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.
Yes, that's the whole idea behind arduino_ci. I'm still trying to wrap my head around it but you can set  and read the states of virtual pins.

In practice I am lazy.
Me too, and what a great feeling it is to kick back and watch Travis CI running thousands of compilations for me.

I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.
Yeah, it's especially nice for open source projects where you get random pull requests. The CI tests automatically run on the pull request so you have this initial triage before you even need to take a look at the proposal. If it doesn't build then the person who submitted the PR can look at the build, find the bug, and then update the PR until it passes the build. A repository maintainer still will need to look at it before it can be merged but that initial bug shakedown can save them a lot of time and effort.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 10, 2018, 02:24 pm
Yeah, it's especially nice for open source projects where you get random pull requests
That's way way beyond my pay grade :)

...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 10, 2018, 08:03 pm
Would it be possible for an author to run tests that verify compatibility with a board that he does not have?
Yes, arduino_ci does this.  

In fact, I was able to develop this system without connecting a single piece of Arduino hardware to my laptop.  "Compatibility with a board" is nothing more than setting the right set of preprocessor defines for the compiler. This in turn causes various features to be enabled or disabled (e.g. serial ports).

You can validate this for yourself by attempting to run serial comms unit tests on a board that doesn't have a serial port (https://travis-ci.org/adafruit/Adafruit_FONA/builds/350460488#L834).


So much of a microprocessor program depends on the physical world outside which cannot be tested using software.
Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.  I provide this power, in a feature that I've termed "GODMODE (https://github.com/ifreecarve/arduino_ci#using-godmode)".  If there's a physical scenario that GODMODE can't replicate, then I'll be glad to update the library appropriately.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 10, 2018, 08:53 pm
Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.
I don't agree with that at all.

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

To be honest, I don't believe using another microprocessor to simulate the external world would be sufficient.


To my mind there is a huge difference between a test system for (say) a program running on a web server where all the input consists of messages from another computer program and a system for testing a microprocessor that controls, or receives data from other physical apparatus.

...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 10, 2018, 10:20 pm
I don't agree [that Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause]

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.
I want you to walk me through this in case I'm missing something painfully obvious.

Let's say that you have Arduino code, and it reads a digital value from a pin.  What possible values can be returned here?  My understanding is either a HIGH, or a LOW -- there are no other possibilities (https://www.arduino.cc/reference/en/language/functions/digital-io/digitalread/). 

So, to properly test your code, you will need two test cases: one for when the value of HIGH is returned, and one for when the value of LOW is returned.  What happens in the physical world is irrelevant, because you've exhaustively covered the possibilities. 

The situation where you're referring to sounds like one where someone makes the assumption that only HIGH or only LOW would always be returned as expected, and only writes one test.  That is not a limitation of the arduino_ci library, it is a failure of the test writer.  Nothing stops them from enumerating all the combinations that the physical world might throw at them.

I challenge you to post a piece of code that I can't properly unit test.  I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 10, 2018, 11:09 pm
I challenge you to post a piece of code that I can't properly unit test.  I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.
Sorry, I know it is lazy of me, but I am not sufficiently interested to go to that much trouble.

Maybe you are right in that the problem lies with the person writing the tests rather than the test software itself.

A common problem that arises in the Forum is switch bounce - I have to say I can't think how I would write a test to prove that my program could deal with that.

Also, regarding switch bounce, it is not always necessary to deal with it. And in some cases the problem does not arise and does not need to be dealt with - and dealing with it when it is not needed just complicates a program and wastes CPU cycles.

I reckon you need a test which first of all determines if there is a switch bounce problem.

A program may work perfectly well with one brand of switch and fail miserably with a different brand.

...R
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: ifreecarve on Mar 12, 2018, 02:03 am
I reckon you need a test which first of all determines if there is a switch bounce problem.
You've phrased this in a strange way.  I'd imagine that every program has a theoretical "minimum acceptable delay between state changes", and whether or not they guard against that (e.g. handling input from a bouncing switch) is entirely testable.  If you purposefully choose not to implement those guards for the purposes of reducing complexity or CPU cycles, then that's your answer -- no test required.

Your concerns about my work are quite valid by the way, and I welcome them.  After all, I'm making some bold claims about what I can test, which run counter to a lot of prevailing wisdom on what is and isn't possible/practical in an Arduino system.  However, without a specific example that shows the problem you describe (i.e. substantiating your claims), your previous comments on that topic just come off as bluster. 
Title: Re: Running automated unit tests on an Arduino library using Travis CI
Post by: Robin2 on Mar 12, 2018, 09:29 am
your previous comments on that topic just come off as bluster. 
I can understand that.

I will remain with the doubters.

...R