Strikes me you have 2 possible hardware problems. Your pot and your servo. You really should test one at a time. I'd eliminate the pot and modify the sketch to feed the servo values to make sure that's good.
If so you know it's an analogue problem with the pot.