Pages: [1] 2   Go Down
Author Topic: unit testing and programming  (Read 3510 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Newbie
*
Karma: 0
Posts: 17
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

is there a way to perform unit and/or functional testing when programming with the arduino platform? if so, what are the tools?
Logged

Left Coast, CA (USA)
Offline Offline
Brattain Member
*****
Karma: 361
Posts: 17293
Measurement changes behavior
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

About the only method presently available is to embed debugging serial write statements within your program and monitor them with the IDE's serial monitor function.

Lefty
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 17
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

as i understand, there are many languages that one may use to program the arduino ucontroller, right? for example, there's C (though it's not called C, on wikipedia, it's referred to as the Arduino programming language). can't we somehow use a C unit testing framework like check (http://check.sourceforge.net/) to test the functions? i am thinking that since C++ is C, we could possibly even use mature C++ unit testing framework (that offers mock objects). i don't know, i'm just thinking out loud now.

debugging by print statements was something i did a long time ago (i do so indirectly with logging frameworks now), but it is definitely looked down upon by computer programmers.

don't you think it is painful to go through a development lifecycle like this? i mean, we won't know what's broken programmatically until we turn on the unit.

how do other frameworks do unit and functional testing of code and/or hardware? thanks.
Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The unit testing frameworks you mention run on the same platform as the code being tested. How do you propose to use them on the Arduino?
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 17
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

pauls, that's the million dollar question i'm asking. smiley-sad

with what's out there and available, have there been any work, thoughts, and/or experiments with unit/functional testing arduino programs/code?
Logged

0
Offline Offline
Faraday Member
**
Karma: 23
Posts: 3480
20 LEDs are enough
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

I do not really get the point of this question. You do not need any fance unit test frameworks for unit testing. The main effort is to implement the tests. Given that the Arduino has only 32k flash your programs can't get that big. So why not just implement the tests straightforward?

If you consider testing the device as a whole during your unit tests then maybe JTAG would be an option. But this is somewhat advanced. I asked about JTAG in this forum before and so far I did not get an answer.

Once I switch to JTAG I will write up my findings though.

Ideally you might want to have a test framework that will controll the device's IO as well from outside. That is a reasonable test framework would require hardware as well as software. Again, sounds like JTAG.
Logged

Check out my experiments http://blog.blinkenlight.net

Sweden
Offline Offline
Jr. Member
**
Karma: 0
Posts: 84
Arduino System Go!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Since most Arduino programs are tightly coupled to the I/O and hardware interaction a test framework must also mock the hardware accurately, so that your tests can include the signals coming in and going out. Currently there is no software Arduino emulator, so you have to take another approach.

 Roughly I think you have the following options:

  • 1 - Implement as much as possible of your logic in ANSI C/C++ libraries and test the libraries standalone.
I find that taking a functional approach, one is able to abstract away from the hardware state and isolate the program logic very well. Using limited state also makes it easier to write tests.

Given that you have verified libraries, you can then just include what you need and tie it together in the main program, which will contain very limited logic.[/list]
  • 2 - Go down the route suggested by Udo Klein
I am not saying that you shouldn't do testing, but embedded programming in limited devices requires a different mindset then the one you use when writing large scale programs. The overhead of an on-board test framework is simply too much. Better then to just implement the tests directly in you code, skipping a framework. Using #ifdef you can then include/exclude test code in your binary.

[/list]
Many bugs occur only when your software is interacting with the hardware, like strangely fluctuating voltages for instance. This means that testing that catch common bugs cannot be limited to software only. Writing robust code for Arduino must include the hardware aspect.

I am an Arduino newbie myself, but I work in the software industry since more than 10 years. Initially I was trying to apply too much of my professional software practices to the Arduino. Now I have realized that this takes a slightly different mindset.

In pure software development you finished artifact is a binary file that executes on generic multi-purpose hardware (like a regular UNIX server). Building with Arduino (or any other microcontroller), your finished artifact is a custom built combination of hardware and software. Your artifact most likely includes a number of external components like servos, LEDs, linkages and power supplies. This means that the software component is often only a small part of your build. The hardware is just as important and most likely more complex.

Testing on the Arduino must therefore not focus only on the software component, as you then loose focus on the whole. Efficient testing should then include the hardware as well, but that is most likely out of scope for most hobbyists.

Test frameworks is mostly used to catch regression as the product evolves. In the case of most Arduino builds, you are doing one-off, or very short series builds. Regression is then less of a problem, which in turn makes automated testing using test harnesses less important. Manual testing is feasible for one-offs and short series.

Sorry for the long post, but I find the subject interesting and I recognize the thoughts reflected in jakester's question.

BTW. Shouldn't this thread be moved to "Development", since it is strictly not a specific trouble shooting question?

/Anders

Logged

Seattle, WA USA
Offline Offline
Brattain Member
*****
Karma: 610
Posts: 49077
Seattle, WA USA
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Initially I was trying to apply too much of my professional software practices to the Arduino. Now I have realized that this takes a slightly different mindset.

I have been a software developer for a long time, too. I think that the skills developed over years of writing software are completely compatible with developing code for the Arduino.

When I am developing a new sketch, I add a few lines of code, and test that. Only when that works do I go on and add more code. That's unit testing, in my mind.

We see a lot of questions on this forum where someone will post 300 lines of code that won't even compile, and ask why not. The code is full of syntax errors, logic errors, commented out code, and plain silly stuff.

A unit testing framework won't help that kind of coder. Unit testing is more of a mind-set, in my opinion, than a fancy set of tools to automate testing.

Creating functions to perform a single task, testing that function, and then leaving it alone when it works, is another professional software practice that gets ignored in Arduino development, far too often. At least, it appears to be from a large number of posts in this forum.
« Last Edit: January 01, 2010, 09:15:21 am by PaulS » Logged

0
Offline Offline
Faraday Member
**
Karma: 23
Posts: 3480
20 LEDs are enough
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

PaulS writes:

>> When I am developing a new sketch, I add a few lines of code, and test that.

And that's exactly the same approach I follow. The basic idea is to try as hard as possible to never have more than 1-2 known bugs at a time and to always have a working version. (This implies a need for version control, something the Arduino IDE misses completely. So I stick to the command line and have my version control.)
Logged

Check out my experiments http://blog.blinkenlight.net

0
Offline Offline
Newbie
*
Karma: 0
Posts: 17
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

i am still consuming the arduino website (it has a lot of pages and documentations, that are all helpful in one way or another). i found references to other similar projects to arduino. in particular, in regards to the sun microcontroller framework/platform, they have an emulator. i think (as mentioned by Anders2009) an emulator may be what i am looking for. i have played with the j2me (java micro, programming for mobile devices) framework, and they have an emulator. you program your code, and then you can perform unit test (with JUnit) and functional test (or QA test) with the emulator. the emulator may be modified to represent certain hardware specs (i.e. screen resolution) from reading your responses, arduino doesn't have an emulator. smiley-sad

pauls, the problem i have with writing a little code, then test, then writing a little code, then test (this type of lifecycle development) is that it is slow (time consuming) and prone to error (manual). your test will have to include uploading, turning on the unit (microcontroller + parts), running functional/regression/QA tests, etc... all of this is manual work and not automated (and anything that involves a human being testing is prone to errors, at least if the human being is me). also, when programming in C, one cannot help but to produce spaghetti code (code that is procedural, or more procedural than declarative), thus the code may not lend itself to unit testing frameworks.

on version control, i don't think the ide needs to have/support that explicitly. you can use CSV or SVN (svn is my choice) to version the code outside of the ide.

as for posting 300+ lines of code, there should be a rule to guide users in posting code. in the java forums, there's the idea of short self-contained correct example (www.sscce.org). i think we may have to promote this idea here too. when i've run into bugs, by preparing a sscce, i sometimes figure out the bug myself and never make it to the forums.

undo klein, you are right, you don't need any fancy unit testing framework, but why reinvent the wheel? the approach you suggested is what a lot of programmers have done in the past until these unit testing frameworks showed up. moreover, these unit testing frameworks (if they are mature) should automate the build, testing, and reporting for you (which i do not want to do, and would rather have something available out there to help me with).

i agree, it will be difficult to help coders that do not write their code to be unit testable (testable in terms of isolated units). but should a coder write unit testable code, we should be able to easily as possible help those coders out (don't you think?).

anders 2009, i like your first suggestion. i think that is probably the way to go given the limitations at the moment. i think abstracting out the hardware is contributes significantly to unit testing of code. in fact, that's what i do in my enterprise programming projects (abstract out the components, i.e. services/facade, that will interact with code units).

i understand what you are saying that once the code is deployed (uploaded), some bug will only arise in the working environment (i.e. fluctuating voltages). for problems like this (and problems related to observational effect), we use logging (which is equivalent to print statements). i wonder if emulators out there are helpful in resolving these types of problems?

also, i agree with what you are saying about hardware vs software testing. testing anything, is complicated (even some unit testing). the catalog of different types of testings are staggering (unit, functional, regression, etc...). but i think in this thread, i was more focused on code testing (in the taste of unit testing).

lastly, i don't know why i posted this thread in this forum, but probably because i thought maybe unit testing (or testing) was related to troubleshooting.

thanks all for the insight. very interesting discussion.
Logged

0
Offline Offline
Faraday Member
**
Karma: 23
Posts: 3480
20 LEDs are enough
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

You write:

pauls, the problem i have with writing a little code, then test, then writing a little code, then test (this type of lifecycle development) is that it is slow (time consuming) and prone to error (manual). your test will have to include uploading, turning on the unit (microcontroller + parts), running functional/regression/QA tests, etc... all of this is manual work and not automated (and anything that involves a human being testing is prone to errors, at least if the human being is me). also, when programming in C, one cannot help but to produce spaghetti code (code that is procedural, or more procedural than declarative), thus the code may not lend itself to unit testing frameworks.

That is you state that this code does not lend to unit test frameworks.

undo klein, you are right, you don't need any fancy unit testing framework, but why reinvent the wheel? the approach you suggested is what a lot of programmers have done in the past until these unit testing frameworks showed up. moreover, these unit testing frameworks (if they are mature) should automate the build, testing, and reporting for you (which i do not want to do, and would rather have something available out there to help me with).

The point of unit test frameworks is indeed often more in helping reporting than in helping detecting bugs. Now I wonder why you would need reporting for a hobby / one person project? For me this would be very much at the end of the wishlist. As already stated: you can write some tests easily without such a framework.
Logged

Check out my experiments http://blog.blinkenlight.net

0
Offline Offline
Newbie
*
Karma: 0
Posts: 17
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

udo klein, what you're saying is wrong regarding unit testing frameworks. they are supposed to help you track bugs first and foremost, and the feature that reports which tests fail is just a natural extension and not non-insignificant part of unit testing.

why do i need testing and reports of those tests for a hobby or one person project? in all my programming projects be they hobby or professional, one or multi-person, i employ unit testing frameworks (among other industry standards). it's just good practice and behavior (good habit).

but you're welcome to be content with the status quo if you want, some of us are asking questions and may even help out in developing this platform in one way or another. cheers.
Logged

0
Offline Offline
Faraday Member
**
Karma: 23
Posts: 3480
20 LEDs are enough
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Jakester: You say I am wrong but you repeat my point.

The unit test frameworks will track bugs not detect them. The detection part is done by the unit tests.

In my opinion the detection part is the hard work. The tracking and reporting is trivial if you are a one person project. You just call all you unit tests and print the output. Unless you expect hundreds of tests to fail on a regular basis you do not desparately need a fancy framework.

If you want to contribute a unit test framework feel free to do so.

Just out of curiosity: what are those other industry standards you apply?
Logged

Check out my experiments http://blog.blinkenlight.net

0
Offline Offline
Faraday Member
**
Karma: 23
Posts: 3480
20 LEDs are enough
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Just in case someone wants to know: my "test framework" is basically Serial.print based. The only twist: whenever I find something that could go wrong I add some definition like so:

Code:
//#define TRACK_SCREENS 1
//#define TRACK_FACTORY 1
//#define TRACK_EVENTS 1
//#define TRACK_FOCUS 1
//#define TRACK_SERIALIZATION 1
//#define TRACK_SCREEN_PERFORMANCE 1

Then I put the following snippets into my code

Code:
     #ifdef TRACK_FOCUS
      Serial.pgm_print(PSTR("next focus: "));
      Serial.println((*screen).focus, DEC);
      Serial.pgm_print(PSTR("current top row: "));
      Serial.println((*screen).top_row, DEC);
      #endif

So whenever I suspect something is wrong with a specific functionality I just remove the // in front of the define and the issue will be tracked.

For things that are really subtle the tracking will always be active.

In order to conserve SRAM in case of heavy tracking I hacked the print library code like so:

Code:
void Print::pgm_print(PGM_P PROGMEM str) {

      uint8_t c;

      while((c = pgm_read_byte(str)) != '\0') {
            this->write(c);
            ++str;
      }
}

void Print::pgm_println(PGM_P PROGMEM str) {

      this->pgm_print(str);
      println();
}
« Last Edit: January 04, 2010, 06:02:09 am by udoklein » Logged

Check out my experiments http://blog.blinkenlight.net

Sweden
Offline Offline
Jr. Member
**
Karma: 0
Posts: 84
Arduino System Go!
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

@Jakester: You might be interested in this thread. Seems a guy have an emulator going, almost all the way. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1261491110/10

@Udo Klein: Your approach is very similar to the one I use myself for Arduino, but it seems like you have taken things a step further than me. I am still a newbie on Arduino. Can you explain the Print::pgm_print(...) stuff a little more in detail and also show how you use it?

As I understand it, you are printing to memory and then you can read that debug info later on?

/Anders
Logged

Pages: [1] 2   Go Up
Jump to: