unit testing and programming

Initially I was trying to apply too much of my professional software practices to the Arduino. Now I have realized that this takes a slightly different mindset.

I have been a software developer for a long time, too. I think that the skills developed over years of writing software are completely compatible with developing code for the Arduino.

When I am developing a new sketch, I add a few lines of code, and test that. Only when that works do I go on and add more code. That's unit testing, in my mind.

We see a lot of questions on this forum where someone will post 300 lines of code that won't even compile, and ask why not. The code is full of syntax errors, logic errors, commented out code, and plain silly stuff.

A unit testing framework won't help that kind of coder. Unit testing is more of a mind-set, in my opinion, than a fancy set of tools to automate testing.

Creating functions to perform a single task, testing that function, and then leaving it alone when it works, is another professional software practice that gets ignored in Arduino development, far too often. At least, it appears to be from a large number of posts in this forum.

PaulS writes:

When I am developing a new sketch, I add a few lines of code, and test that.

And that's exactly the same approach I follow. The basic idea is to try as hard as possible to never have more than 1-2 known bugs at a time and to always have a working version. (This implies a need for version control, something the Arduino IDE misses completely. So I stick to the command line and have my version control.)

i am still consuming the arduino website (it has a lot of pages and documentations, that are all helpful in one way or another). i found references to other similar projects to arduino. in particular, in regards to the sun microcontroller framework/platform, they have an emulator. i think (as mentioned by Anders2009) an emulator may be what i am looking for. i have played with the j2me (java micro, programming for mobile devices) framework, and they have an emulator. you program your code, and then you can perform unit test (with JUnit) and functional test (or QA test) with the emulator. the emulator may be modified to represent certain hardware specs (i.e. screen resolution) from reading your responses, arduino doesn't have an emulator. :frowning:

pauls, the problem i have with writing a little code, then test, then writing a little code, then test (this type of lifecycle development) is that it is slow (time consuming) and prone to error (manual). your test will have to include uploading, turning on the unit (microcontroller + parts), running functional/regression/QA tests, etc... all of this is manual work and not automated (and anything that involves a human being testing is prone to errors, at least if the human being is me). also, when programming in C, one cannot help but to produce spaghetti code (code that is procedural, or more procedural than declarative), thus the code may not lend itself to unit testing frameworks.

on version control, i don't think the ide needs to have/support that explicitly. you can use CSV or SVN (svn is my choice) to version the code outside of the ide.

as for posting 300+ lines of code, there should be a rule to guide users in posting code. in the java forums, there's the idea of short self-contained correct example (www.sscce.org). i think we may have to promote this idea here too. when i've run into bugs, by preparing a sscce, i sometimes figure out the bug myself and never make it to the forums.

undo klein, you are right, you don't need any fancy unit testing framework, but why reinvent the wheel? the approach you suggested is what a lot of programmers have done in the past until these unit testing frameworks showed up. moreover, these unit testing frameworks (if they are mature) should automate the build, testing, and reporting for you (which i do not want to do, and would rather have something available out there to help me with).

i agree, it will be difficult to help coders that do not write their code to be unit testable (testable in terms of isolated units). but should a coder write unit testable code, we should be able to easily as possible help those coders out (don't you think?).

anders 2009, i like your first suggestion. i think that is probably the way to go given the limitations at the moment. i think abstracting out the hardware is contributes significantly to unit testing of code. in fact, that's what i do in my enterprise programming projects (abstract out the components, i.e. services/facade, that will interact with code units).

i understand what you are saying that once the code is deployed (uploaded), some bug will only arise in the working environment (i.e. fluctuating voltages). for problems like this (and problems related to observational effect), we use logging (which is equivalent to print statements). i wonder if emulators out there are helpful in resolving these types of problems?

also, i agree with what you are saying about hardware vs software testing. testing anything, is complicated (even some unit testing). the catalog of different types of testings are staggering (unit, functional, regression, etc...). but i think in this thread, i was more focused on code testing (in the taste of unit testing).

lastly, i don't know why i posted this thread in this forum, but probably because i thought maybe unit testing (or testing) was related to troubleshooting.

thanks all for the insight. very interesting discussion.

You write:

pauls, the problem i have with writing a little code, then test, then writing a little code, then test (this type of lifecycle development) is that it is slow (time consuming) and prone to error (manual). your test will have to include uploading, turning on the unit (microcontroller + parts), running functional/regression/QA tests, etc... all of this is manual work and not automated (and anything that involves a human being testing is prone to errors, at least if the human being is me). also, when programming in C, one cannot help but to produce spaghetti code (code that is procedural, or more procedural than declarative), thus the code may not lend itself to unit testing frameworks.

That is you state that this code does not lend to unit test frameworks.

undo klein, you are right, you don't need any fancy unit testing framework, but why reinvent the wheel? the approach you suggested is what a lot of programmers have done in the past until these unit testing frameworks showed up. moreover, these unit testing frameworks (if they are mature) should automate the build, testing, and reporting for you (which i do not want to do, and would rather have something available out there to help me with).

The point of unit test frameworks is indeed often more in helping reporting than in helping detecting bugs. Now I wonder why you would need reporting for a hobby / one person project? For me this would be very much at the end of the wishlist. As already stated: you can write some tests easily without such a framework.

udo klein, what you're saying is wrong regarding unit testing frameworks. they are supposed to help you track bugs first and foremost, and the feature that reports which tests fail is just a natural extension and not non-insignificant part of unit testing.

why do i need testing and reports of those tests for a hobby or one person project? in all my programming projects be they hobby or professional, one or multi-person, i employ unit testing frameworks (among other industry standards). it's just good practice and behavior (good habit).

but you're welcome to be content with the status quo if you want, some of us are asking questions and may even help out in developing this platform in one way or another. cheers.

Jakester: You say I am wrong but you repeat my point.

The unit test frameworks will track bugs not detect them. The detection part is done by the unit tests.

In my opinion the detection part is the hard work. The tracking and reporting is trivial if you are a one person project. You just call all you unit tests and print the output. Unless you expect hundreds of tests to fail on a regular basis you do not desparately need a fancy framework.

If you want to contribute a unit test framework feel free to do so.

Just out of curiosity: what are those other industry standards you apply?

Just in case someone wants to know: my "test framework" is basically Serial.print based. The only twist: whenever I find something that could go wrong I add some definition like so:

//#define TRACK_SCREENS 1
//#define TRACK_FACTORY 1
//#define TRACK_EVENTS 1
//#define TRACK_FOCUS 1
//#define TRACK_SERIALIZATION 1
//#define TRACK_SCREEN_PERFORMANCE 1

Then I put the following snippets into my code

      #ifdef TRACK_FOCUS
      Serial.pgm_print(PSTR("next focus: "));
      Serial.println((*screen).focus, DEC);
      Serial.pgm_print(PSTR("current top row: "));
      Serial.println((*screen).top_row, DEC);
      #endif

So whenever I suspect something is wrong with a specific functionality I just remove the // in front of the define and the issue will be tracked.

For things that are really subtle the tracking will always be active.

In order to conserve SRAM in case of heavy tracking I hacked the print library code like so:

void Print::pgm_print(PGM_P PROGMEM str) {

      uint8_t c;

      while((c = pgm_read_byte(str)) != '\0') {
            this->write(c);
            ++str;
      }
}

void Print::pgm_println(PGM_P PROGMEM str) {

      this->pgm_print(str);
      println();
}

@Jakester: You might be interested in this thread. Seems a guy have an emulator going, almost all the way. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1261491110/10

@Udo Klein: Your approach is very similar to the one I use myself for Arduino, but it seems like you have taken things a step further than me. I am still a newbie on Arduino. Can you explain the Print::pgm_print(...) stuff a little more in detail and also show how you use it?

As I understand it, you are printing to memory and then you can read that debug info later on?

/Anders

void Print::pgm_print(PGM_P PROGMEM str) {

      uint8_t c;

      while((c = pgm_read_byte(str)) != '\0') {
            this->write(c);
            ++str;
      }
}

void Print::pgm_println(PGM_P PROGMEM str) {

      this->pgm_print(str);
      println();
}

Is code that I have added to the standard Print library. This gives me two new methods pgm_print and pgm_println that will print strings from progmem instead of sram. Since the hardware serial class inherits from Print it inherits these methods as well. Hence Serial.pgm_print will print progmem strings to the serial interface.

The PSTR macro is a standard macro of AVR LIBC. It defines a progrmem string in place.

An example was already provided:

      #ifdef TRACK_FOCUS
      Serial.pgm_print(PSTR("next focus: "));
      Serial.println((*screen).focus, DEC);
      Serial.pgm_print(PSTR("current top row: "));
      Serial.println((*screen).top_row, DEC);
      #endif

This does the following: if TRACK_FOCUS is defined the 4 debug lines will be compiled into the code. Instead of Serial.pgm_print(PSTR("next focus: ")); I could write Serial.print("next focus: "); However the first statement will put the string into progmem and the second one will put it into sram.

So I do not write debug information into memory. If I would try to do this I would write debug information into eeprom and read it with a programmer later on. Writing debug information into progmem would be theoretically possible but this is something I would rather avoid until I have absolutely no other option.

udo: The unit test frameworks will track bugs not detect them. The detection part is done by the unit tests.

a unit test framework will help you write unit tests, no? that is one part of many things that a unit test framework can help you with. in fact, a unit test framework will help you detect bugs more than "track" them. a bug tracking system is complementary to a unit test framework (i.e. use junit for unit testing and as a unit testing framework, use bugzilla/jira to track bugs).

udo: In my opinion the detection part is the hard work. The tracking and reporting is trivial if you are a one person project. You just call all you unit tests and print the output. Unless you expect hundreds of tests to fail on a regular basis you do not desparately need a fancy framework.

tracking and reporting bugs is non-trivial. have you looked at frameworks and web applications dealing with bug tracking and/or reporting? you're arguing that these items/tools/practices are not relevant to a "one person project", and, in my opinion, that is wrong. i have taken on many contracting positions involving "one person" (just me) in software engineering, and i do use bug tracking software, unit testing framework, version control, etc...

udo: If you want to contribute a unit test framework feel free to do so.

i might need to given your helpful comments and code here. :slight_smile:

udo: Just out of curiosity: what are those other industry standards you apply?

in regards to software engineering: unit testing, design patterns (mvc, ioc, fc, etc...) , agile development (xp), version control, continuous integration, etc...

cheers.

@jakester
In all the work you've done, the development has been on the same platform that the executable is to be run on, right?

When developing code for the Arduino, the development is done on the PC, and the execution is done on the Arduino.

That is a fundamental change, and one that does not lend itself to unit testing within a test framework. The test framework executes the code on the same platform it is running on. It would be very difficult to fit a unit test framework and bug tracking system on the Arduino, along with the code to be executed.

But, if you're able to pull it off, that would be great. Keep us posted on your progress.

Absolutely correct. But given the fact that some people are already in developing AVR emulators in software it would be cool to have a test framework that emulates and tracks everything. Since the processors are small enough even omniscient debugging Debugger - Wikipedia would then be feasible. This would be really extraordinary. However this would be much more work than just some unit test framework :slight_smile:

pauls, that's not unit testing anymore. i think you're walking into the realm of functional testing then (when you're executing on the runtime platform). are we all talking about the same thing here?

ideally, code should be tested, and best practices suggest at a "unit" level (your coding should produce fine and self-contained units of logic). theoretically, regardless of programming language, code is unit testable (complicated, as suggested in this thread, by programming habits).

even if you code in java and test using a java unit testing platform, successful testing on a developer machine will not mean necessarily that on a server machine the code will produce the same results (read the specs on jvm in client/server mode). meaning, writing and unit testing code on what is supposed to be the "same platform" in development and deployment do not make bug detection any easier (than if the environments were different).

at any rate guys, thanks for all the help. i think i've got enough to go on and workaround the limitations. you've all been very helpful and kind.

cheers.

We are using Arduino boards for data acquisition in a large scientific experiment. Subsequently, we have to support several Arduino boards with different implementations. I wrote python utilities to dynamically load Arduino hex images during unit testing. The code found on the link below supports Windows and OSX via configuration file. To find out where your hex images are placed by the Arduino IDE, hit the shift key before you hit the build (play) button. Hit the shift while hitting upload to find out where your avrdude (command line upload utility) is located on your system / version of Arduino. Alternatively, you can look at the included config files and use your install location (currently on Arduino 0020).

That post above was my first post and it wouldn't let me include a link. Here is the link to the python unit testing / loading code.

In my day job, I am a great fan of test-driven development, but one of the things I love about the Arduino is that I can have it do all manner of fancy things in a few dozen lines of code.

I don't need unit testing because its all so delightfully concise and simple.

You need it when your building systems with 100,000 lines of code for a machine with 4GB.

2K and worried about TDD -- I don't think so :wink:

I don't need unit testing because its all so delightfully concise and simple.

The fun stops when one single line of code produces strange effects sometimes and you have no clue if it's a resource conflict, something in the hardware or your software.

And to top it off, whenever you try to debugging, you end up measuring the effects introduced by your debugging probe and not the problem.

Real men debug with oscilloscopes and chainsaws.

Korman

Real men do not debug at all :sunglasses:

AlphaBeta, you have a point here.

Chuck Norris doesn't debug - all bugs flee in terror when he turns on the power.

Korman