The rate gyro is the source of the drift, and it must be calibrated.
Since the drift is temperature sensitive, the calibration will never be perfect. For a stable 3D orientation, you need a 9DOF sensor that includes a horizontal reference (e.g. magnetometer).