If you want to understand how to do this stuff, you have to learn that math. Otherwise, ask or hire someone to write it for you. The HMC5883L is no longer manufactured, so you might have a tough time doing that.
Adafruit has a calibration tutorial and ready made code on their web site, for the magnetometers they sell, so you don't need to write anything (just buy their product).
Edit: it is possible that Adafruit's calibration code supports the HMC5883L, which they used to sell. Take a close look at it and/or inquire on the Adafruit forum. Magnetometer Calibration | Adafruit SensorLab - Magnetometer Calibration | Adafruit Learning System