They are called decoupling (or bypass) capacitors and are used to suppress noise.
The choice of the size will depend on the application and the power supply.
0.1 uF is usually a good size to start with, but if your circuit is still misbehaving, you can put a larger one in parallel.
As for their importance, one of our more prolific members here (sorry, I forget names), says something along the lines of
"When an otherwise good looking circuit misbehaves, 9 out of 10 times the problem is poor decoupling. The remaining 1/10 will eventually turn out to be decoupling as well."
Author, sorry for butchering the quote.