But the noise goes away (even when still reading the pot) at the max and min settings of the pot.
In which case have you tried putting a serese resistor of about 20R between the top of the pot and +5V. Then put a 47uF capacitor from top to bottom of the pot. Finally put a 1uF capacitor from the wiper to ground. This would reduce the electrical noise from the pot and give a more stable reading.
Have you tested how much noise you have? Just read the pot and print it out, what sort of variations do you see? You could also try a running average on the readings from the pot. Any jitter on the delay will cause noise due to an inconsistent sample rate.