It seems that the forum software ate my previous reply.
So the next thing to look at is the sweep example that comes with the servo library. The delay between steps sets the speed. Vary the delay time with the pot reading. Use the map function to scale the 0 to 1023 from the pot to the range of delay times.
Then learn to use millis() for timing and get rid of delay.
Non-blocking timing tutorials:
Several things at a time.
Beginner's guide to millis().
Blink without delay().