The most likely cause is that there is a DIV8 Fuse that is ON BY DEFAULT. If you don't remember to turn it off, or if you don't program the fuses at all, you will run at much lower clock rate than you expect.
The "Burn Bootloader" process of Arduino sets the fuses to the correct state for the Arduino hardware (since it pretty much matches your hardware, you lucked out!)