In a rc boat, I use a compass + GPS sensor. I calculate the angle of the compass in relation to the angle of the target point. When the compass is perfectly calibrated everything is ok. Problems begin when the compass mounts eg 10 or 20 degrees in a different direction, or when the compass is incorrectly calibrated and deflects by 10-20 degrees left or right. I deal with it like this:
Subtracts the heading from the gps module (nmea with GPGGA) from the compass heading, that is:
error = (heading_gps - heading_compass)
I am doing a compass correction:
new_bearing = (heading_compass + error) % 360
The effect is that when the error is a few degrees it is ok, but when it is 30-40 error it is too big a correction in relation to the target point.