Rerun magneto with a much small value (e.g. 100) for the "norm of the magnetic or gravitational field", so that you get more significant figures in the correction matrix A-1.
Then put the 12 numbers from magneto into the code.
If you have used the "raw values" from the sensor as instructed, the scale factors (100000.0/1100.0), etc. below SHOULD NOT BE USED. Take them out of the code. They never should have been there in the first place.
compass.m.x*(100000.0/1100.0)
Rather than waste your time trying to correct the mistakes made by the author of the post linked in reply #6, you would be better off with this post and the examples attached to it.