I'm not familiar with the hardware you describe.
But I know it's normal for robot vehicles to drift away from straight ahead if they don't have any feedback to tell them when they are not going straight ahead. Just turning the wheels at what should be the same speed with no feedback mechanism won't work and drift will be inevitable.
So what feedback is available for your robot to know if it is going straight?
Encoders on the wheels is a common method, but not perfect when the wheels can slip.
A magnetometer is another technique but is also not perfect, especially in buildings with steel frames or where other large metallic objects are nearby.