tilt compensated magnetometer

Are the mag and accl mounted horizontally (maximum rotation never carries it past vertical)?

Has the mag been calibrated to account for the hardware you are using?

Post your debug outputs so we can see what you're doing.