Go Down

Topic: Running automated unit tests on an Arduino library using Travis CI (Read 3387 times) previous topic - next topic

Robin2

I think the entire Arduino community would benefit tremendously from easier automated testing of 3rd party projects. One of the best things about Arduino is the huge number of libraries and hardware packages available but it can be a bit hit and miss.
I have not come across libraries that simply don't work. My experience with libraries and from reading about Forum members using libraries is that there are two main problems
  • Totally inadequate documentation
  • Incompatibility between libraries.

I don't believe unit testing can address either of those problems.

To some extent it may never be possible to eliminate all incompatibilities between libraries simply because there is a very limited set of hardware that they must all share.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.
I think that's the case for this project, but if you try it out and believe otherwise then I would welcome your feedback.

pert

I have not come across libraries that simply don't work.
I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.

Obviously if the ability to run unit-tests can be included without reducing the simplicity of the Arduino system I have no objection to the facility being there.
Agreed. The GUI that the average user interacts with must be kept very simple. But the Arduino IDE can allow for advanced usage under the hood. Regarding the test folder thing (which is the only specific change that's been mentioned) I'm not sure it would even end up as a change to the current IDE code but only a change in the text of the library specification. Currently the Arduino IDE has a source folder whitelist approach, meaning that there shouldn't be any problem putting the unit tests in the test folder. I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?

I suppose the test folder being in the root of the library is an existing convention outside the Arduino world?
Yes, that's accurate.  In Java, there would be a "src" directory for the code and "test" directory for the tests.  In Python the code would live in a directory that's the same as the library name, and the tests in "test".  Ruby would have a "lib" and a "spec" directory, and JavaScript a "src" and "test" directory.   In Swift, code would be split among a number of directories but tests live in "Tests".  Etc.

I don't think it makes any sense to clutter the Arduino IDE.  Adding a whitelist entry for "test" to the library specification would be best-case-scenario for me, including (and/or especially) if they didn't prescribe a particular test tool to manage that.  It would be up to the folks who want to do collaborative development via GitHub to implement their own tests and CI in that directory.

Robin2

I do, but I look at a ton of Arduino libraries and specifically look for bugs. I think if you use standard AVR boards and popular libraries you are much less likely to encounter bugs simply because so many people are using the library they become the unit tests.
I suspect part of the problem is that the authors of libraries don't have access to a wide range of Arduino boards.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?


I think I would be content with decent documentation that, inter alia, lists the boards that the author has tested the library on. And I don't mean unit tests.

If people can't be bothered to write documentation what chance is there that they will take the trouble to learn how to do unit testing.

...R

Two or three hours spent thinking and reading documentation solves most programming problems.

pert

Well I think one reason developers don't like writing documentation is because, even though they're sharing it, the code is primarily written for themselves and they think (albeit wrongly) of documentation as being an extra task solely for the benefit of other users. However, as a user of the code, there is an obvious direct benefit of having it be thoroughly and efficiently tested.

Automated testing also makes writing code much more enjoyable because you get to do just the fun, creative part that humans are best at and leave all the boring stuff that machines are best at to the computer.

The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way. In fact that's how I've been with unit testing. I've been thinking about it a lot lately but there hasn't been a clear path to actually getting going with it. Maybe arduino_ci will help me finally get there.

I've been working on making it easier to do continuous integration compilation tests of Arduino projects with the hopes that if that initial barrier is lowered then more people will start doing it. So far I don't think I've made any difference but I'll keep working at it.

Robin2

The problem is there is this steep learning curve. In the time you spend initially getting a testing system set up you could have just manually done the tests several times over. Of course making the effort will pay off many times over in time but people often don't think about things that way.
I don't reckon I do enough Arduino programming to get the pay-off that you describe.

Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.

So much of a microprocessor program depends on the physical world outside which cannot be tested using software. And building another microprocessor to simulate the real world is fraught with difficulties, even if there was time to do so.


You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.


In theory I can see a lot of value in test-driven-development  -  building the test before you write the code. In practice I am lazy.


I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.


...R
Two or three hours spent thinking and reading documentation solves most programming problems.

pert

Setting up the tests is a considerable chore when you start a new project, even if you are familiar with how to do it.
Yes, my understanding is that it is a good bit of work writing unit tests. It shouldn't be too bad if you write them as you go along but for existing projects it's going to take a real effort. Even so, it's not all or nothing. You can start with the highest priority tests and add coverage gradually over time.

The compilation tests are a much lower hanging fruit. In my case it mostly consists of having Travis CI use the Arduino IDE to compile library examples. If the examples compile then the build passes. If one of them doesn't compile then the build fails. The trouble is that doesn't tell you whether the code actually works, only that it compiles. I found there were still some tricky parts to doing those compilation tests with Travis CI in the way I wanted so I ended up wrapping all that up in a Bash script, which makes it even easier to set up that sort of continuous integration. It saves a report of the results of the compilations for each build, which makes it easy to compare changes between commits, such as checking if the flash or global SRAM memory usage changed. I can also compile with multiple versions of the Arduino IDE for multiple boards to make sure no incompatibilities were introduced.

I suspect the problem that will bite you in the ass is the one that you never thought of building a test for.
Sure, but then afterwards you can write a test for it so it will never bite you again.

You have not said if unit testing could allow someone to verify code for hardware they don't have. If that were possible I could see a lot of value.
Yes, that's the whole idea behind arduino_ci. I'm still trying to wrap my head around it but you can set  and read the states of virtual pins.

In practice I am lazy.
Me too, and what a great feeling it is to kick back and watch Travis CI running thousands of compilations for me.

I can certainly see the value of automated testing for a project that involves several programmers because it should eliminate the risk that programmerA inadvertantly breaks programmerB's code. But in that case you would build into the budget the cost of building the test system.
Yeah, it's especially nice for open source projects where you get random pull requests. The CI tests automatically run on the pull request so you have this initial triage before you even need to take a look at the proposal. If it doesn't build then the person who submitted the PR can look at the build, find the bug, and then update the PR until it passes the build. A repository maintainer still will need to look at it before it can be merged but that initial bug shakedown can save them a lot of time and effort.

Robin2

Yeah, it's especially nice for open source projects where you get random pull requests
That's way way beyond my pay grade :)

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Would it be possible for an author to run tests that verify compatibility with a board that he does not have?
Yes, arduino_ci does this.  

In fact, I was able to develop this system without connecting a single piece of Arduino hardware to my laptop.  "Compatibility with a board" is nothing more than setting the right set of preprocessor defines for the compiler. This in turn causes various features to be enabled or disabled (e.g. serial ports).

You can validate this for yourself by attempting to run serial comms unit tests on a board that doesn't have a serial port.


So much of a microprocessor program depends on the physical world outside which cannot be tested using software.
Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.  I provide this power, in a feature that I've termed "GODMODE".  If there's a physical scenario that GODMODE can't replicate, then I'll be glad to update the library appropriately.

Robin2

Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause.
I don't agree with that at all.

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.

To be honest, I don't believe using another microprocessor to simulate the external world would be sufficient.


To my mind there is a huge difference between a test system for (say) a program running on a web server where all the input consists of messages from another computer program and a system for testing a microprocessor that controls, or receives data from other physical apparatus.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

I don't agree [that Testing the physical world is irrelevant if you have the power to (via software) put the system into any state that the physical world might cause]

It assumes that the physical world is always properly HIGH or LOW and that it properly changes state at the time when you expect it to. At the very least the microprocessor program should be able to deal with the situation when there is a fault in the physical world it is attached to.
I want you to walk me through this in case I'm missing something painfully obvious.

Let's say that you have Arduino code, and it reads a digital value from a pin.  What possible values can be returned here?  My understanding is either a HIGH, or a LOW -- there are no other possibilities

So, to properly test your code, you will need two test cases: one for when the value of HIGH is returned, and one for when the value of LOW is returned.  What happens in the physical world is irrelevant, because you've exhaustively covered the possibilities. 

The situation where you're referring to sounds like one where someone makes the assumption that only HIGH or only LOW would always be returned as expected, and only writes one test.  That is not a limitation of the arduino_ci library, it is a failure of the test writer.  Nothing stops them from enumerating all the combinations that the physical world might throw at them.

I challenge you to post a piece of code that I can't properly unit test.  I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.

Robin2

I challenge you to post a piece of code that I can't properly unit test.  I win either way -- either I can successfully test it with my library as-is, or I can use your example to improve arduino_ci.
Sorry, I know it is lazy of me, but I am not sufficiently interested to go to that much trouble.

Maybe you are right in that the problem lies with the person writing the tests rather than the test software itself.

A common problem that arises in the Forum is switch bounce - I have to say I can't think how I would write a test to prove that my program could deal with that.

Also, regarding switch bounce, it is not always necessary to deal with it. And in some cases the problem does not arise and does not need to be dealt with - and dealing with it when it is not needed just complicates a program and wastes CPU cycles.

I reckon you need a test which first of all determines if there is a switch bounce problem.

A program may work perfectly well with one brand of switch and fail miserably with a different brand.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

I reckon you need a test which first of all determines if there is a switch bounce problem.
You've phrased this in a strange way.  I'd imagine that every program has a theoretical "minimum acceptable delay between state changes", and whether or not they guard against that (e.g. handling input from a bouncing switch) is entirely testable.  If you purposefully choose not to implement those guards for the purposes of reducing complexity or CPU cycles, then that's your answer -- no test required.

Your concerns about my work are quite valid by the way, and I welcome them.  After all, I'm making some bold claims about what I can test, which run counter to a lot of prevailing wisdom on what is and isn't possible/practical in an Arduino system.  However, without a specific example that shows the problem you describe (i.e. substantiating your claims), your previous comments on that topic just come off as bluster. 

Robin2

your previous comments on that topic just come off as bluster. 
I can understand that.

I will remain with the doubters.

...R
Two or three hours spent thinking and reading documentation solves most programming problems.

Go Up