dear Magician,
I am afraid I am not that good in electronics

. The magnetometer is from alphalab and quite new:
http://www.trifield.com/content/dc-milligauss-meter-3-axis/I will try to answer your questions
I'm not worry about -2V, hope you read Atmega168 data regarding analog inputs voltage limits.
What I try to clear is:
- how device powered, +5, +3.3 or something else?
My AT MEGA is powered at +5 V. I used 3.3 V as a "calibration value" I think this is a clearer definition of the 3.3 V issue than reference value. (look at my reply to retrolefty).
The magnetometer has a transformer form 220 V AC to 12 V DC
- are the on-board of magnetometer any voltage regulator IC?
I am sorry, I do not know what "IC" means. I can send an email to alphalab to ask them about that details.
- does output signals get affected by changing power supply voltage?
Chart Vout / Vpow would be very helpful.
Well, as I said, I checked the Vout of the magnetometer with other instrumentation, and I got the same results as in the display of it without changing anything (not even the socket where the transformer was plugged).
- what output resistance/impedance?
That is a good question. I measured it (not powered) and it was of 80 kOhm, but I do not know if that is the way of measuring it, or if I should directly ask the company.
- what is wide-band in Hz?
No idea, but I will take advantage to learn something new from you: what is that and why is this important?
When you say accuracy 0.5% , what voltage/temperature/load_resistance range it is true for?
well, it is true at about 23 °C (typical Lab temperature). Arduino values where different from the (correct) display and multimeter values at the same time and in the same room, thus it was at the same temperature. The load resistance should be the same as for oscilloscopes and multimeters, I do not know how much it is, but quite high, I suppose. I believed that the load resistance is the problem, but I increased it to 10 MOhm with no different results (I just put a resistance of 10 MOhm at the input of arduino)...
Can you read marking on the chips of magnetometer, so I can try to download datasheet from net?
I am sorry, I will not open the magnetometer because it is still in warranty... I could ask alphalab, but I am not sure if they would tell me that...