Imahilus, - isn't that a cracker, not a hacker?
The only people that know the difference are the "real" hackers. The general public and the crackers themselves don't often differentiate.
I'm with cowjam's way of saying it. I tend to tell people "I like to hack together bits of hardware in my spare time." It generally gets the question of "what do you mean by that?" If I said: "I like hacking" people generally tightly grip their cell phones and slowly walk away.
On a related note, I watched the movie "Hackers" this weekend. I almost had an aneurism. "Run Anti-Virus!!"