3ms is less than an hour. Just thought I'd point that out.
I find myself in the same situation as the original poster sometimes: I have to wait for the whole sketch to recompile because I forgot to plug in the board. Is there any way to get the basic Arduino IDE to only compile/link the parts that have changed? This would also be useful for compiling sketches with multiple files: if you haven't changed the file, it doesn't need to recompile.