Yes, that is "a lot" and something is wrong. Maybe the pot is wired wrong... Sometimes the slider is not physically the center contact.
What's the value of the pot? (Higher values tend to pick-up more noise.)
What happens at the minimum & maximum rotation?
You're describing an analog input. 