Rather than use the value from the potentiometer directly, use it as a max or min setting and use another variable to control the speed of the motor.
Then at whatever intervals seem reasonable (maybe every 100 millisecs) adjust the motor speed variable up or down in steps until it matches the value from the potentiometer. Something like this
byte adjPWM = 5; // just a guess for illustration
unsigned long changeIntervalMillis = 100;
if (millis() - prevChangeMillis >= changeIntervalMillis) {
prevChangeMillis += changeIntervalMillis;
if (motorPWM > potPWM) {
motorPWM -= adjPWM;
}
else if (motorPWM < potPWM) {
motorPWM += adjPWM;
}
analogWrite(motorPin, motorPWM;
}
...R