I'm not entirely clear with the exact problem you're having, except that you want to control the servo based on the readings from a compass, and you're having issues with the algorithm you're using. The problem seems to stem from the idea that you don't want big swings from the servo when you "shift around the poles", but there's no way, with a servo, to move from 180 to 0 without going back all the way around.
You can stop movement when you reach the limit traveling one direction (i.e., if the differences between values read from the compass indicate that you're moving clockwise, don't rotate past 180, and don't rotate past 0 when moving counter-clockwise.), but you would never be able to move in a full-circle using this method.
The real question is: are you attempting to control the absolute position of the servo, or the direction you want it to move, from the compass readings?
If all you want to do is make the camera move right when you turn clockwise (presuming values go up when you rotate clockwise), then just record the difference between your readings to determine if you're moving clockwise (positive change) or counter-clockwise (negative change). Then, for every two degrees of change in the compass, move the servo a single degree in the appropriate direction until you reach the end of possible movement. (You have to do two compass degrees for one servo degree, as the servo likely maxes out at 180, whereas the compass has 360 degrees.) When you reach the end of movement, you ignore any changes in the compass readings until they reverse direction.