I don't recommend using a potentiometer.
My guess is that the SN754410 is damaged and that is why the speeds are different. An easy way to check is to swap the leads to the motor, and if the motor spins faster while going backward then it is the driver. I've burnt up plenty of motor drivers in my past.