I am wanting my sensors to give me a full 6 DOF, or I guess 9 DOF with the magnetometer, so I assume I need to properly calibrate all 3 axis of the acc and mag. I have tried calibrating by rotating 360 degrees around each axis, and then translating in a circle in all 3 planes.
I am using Fabio Varesano's (may his soul rest in peace) calibration tool which is in Python and uses an ellipsoid into sphere algorithm.
http://www.varesano.net/blog/fabio/ellipsoid-sphere-optimization-using-numpy-and-linalg
Maybe the algorithm used is more sensitive than your Matlab code with respect to needing smooth data in all planes and axes of rotation.
I think I will probably create a gimbal like I mentioned above, it will be fun simple project anyway, and if I can't calibrate my sensors with that, than I will probably look into other algorithms or maybe writing my own to calibration program for my sensors.
Thanks for the help!