When the judge asked the Locksmith what he was doing when he was arrested he said "Making a bolt for the door"
One way a stepper motor can "drift" is if it misses steps because it is being driven too fast or if it does not have enough torque to move the load. And torque reduces with speed.
You need to post a link to the datasheet for the stepper motor and tell us what stepper motor driver and stepper power supply (volts and amps) you are using.