That's the "trick". You can slow it down by moving in small-steps. (I don't know if the library allows fractional degrees.) You can't speed it up.
You should probably start by writing some code to move it slowly before trying to control it with a pot.