Stabilise potentiometer readings?

The accuracy of the ADC of the ATMega chips is already +- 2. Also, the linearity of an inexpensive potentiometer is around 5%. I don't think you can improve what you have.

If you need more accurate values you would need to move to an optical encoder.