See:
7 dpost #7
If you use a factory new ATMEGA328P, you have to "Arduinoize" it by burning a bootloader on to it. As a side effect, this sets the fuses which make it compatible with the Arduino environment.
If, however, you are not using a bootloader, and use an Arduino core which supports this, then you have to use AVRDUDE to set the fuses which have an impact on the timing.
Beware of using an ICSP programmer to load a sketch if there has been a bootloader previously on the chip without previously cleaning the fuses.