I'm a bit at my wits' end here - I'm trying to build a tilt compensated compass for my autonomous sailboat (ardusailor!). I'm using an InvenSense MPU9150. Originally, I used the built-in fusion support on the sensor to get a quaternion, pull the yaw/pitch/roll angles from that, and then use this formula to do the tilt compensation:

float heading = atan2(-(mz * s_phi - my * c_phi), mx * c_theta + my * s_theta * s_phi + mz * s_theta * c_phi);

where the various s_angle is sin(angle) and c_angle is cos(angle). That didn't work. I tried using a vector-based approach stolen from here. That didn't work. Then, I took away the tilt compensation, and just did an uncompensated atan2(Yh,Xh), and that produced very strange result as well.

Basically, as I rotate the sensor about the z axis, the value rotates between 70 and -10 degrees, completing a full circle (i.e. as i make a 360 degree rotation, it starts at 70, gets to -10, and then back up to 70). 70 is at about 0* magnetic, 10 is at about 180, 0 is at about 70-80.

I see the same behavior from an HMC5883L magnetometer chip as well. The thing is, looking at raw values, I get magnetic values that seem fine, and hard and soft iron offsets are in place:

top row is corrected for offsets (using an ellipsoid fit method), bottom is raw. The numbers may look skewed, but they aren't - the scales aren't all the same. Graphs are, in order, x:y, y:z, x:z

What could this be?