Code works on boot loader but not when directly programmed

Part of "burning the bootloader" is setting the fuses to change the system clock from the factory default (8 MHz internal divided by 8 to get 1MHz) to the settings Arduino uses (external 16 mHz crystal). Could that be the problem?