Thank you! I had just discovered that. I do not know how to set the fuses, or what they should be set to work with the ceramic resonator I'm using. I'm quite sure that is the issue. I'm programming the board directly not bootloading it. The data sheet has different settings:
Device clocking option CKSEL3..0
Low power crystal oscillator 1111 - 1000
Full swing crystal oscillator 0111 - 0110
Low frequency crystal oscillator 0101 - 0100
Internal 128kHz RC oscillator 0011
Calibrated internal RC oscillator 0010
External clock 0000
Reserved 0001
It's still all greek to me as I don't know how to set the fuses. never had to mess with it before.