As said, pots MUST be powered from the same supply as the Arduino A/D.
And "supply" means two wires. Power and ground.
So the pots must have dedicated (not shared) supply and ground to the Arduino to be stable and independent of other power users. No way around that.
A pot (voltage divider) outputs a ratio of it's voltage on the wiper.
If you power the pot with 5volt, and turn it to the center position, then it outputs 2.5volt.
But if the supply suddenly drops to 4.8volt, then the wiper voltage drops to 2.4volt (same 50% ratio).
The A/D has the same behaviour, but opposite.
Half of the supply on the input always returns 512, but 2.5volt does not always return 512.
Because of the same and opposite behaviour of pot and A/D, returned values stay the same with supply variations.
That is, if you power both from the same supply.