It sounds like you could be getting the wrong startup code, with vectors in the wrong place.
What EXACTLY does your build command end up looking like (the gcc command that make or cmake produces)? And what version of avr-libc do you have? (and/or post the disassembly of your test program...)