Have a look into the datasheet of the ATmega at the ADC section. There is a Analog Input Circuitry image. Before the ADC can convert the voltage an internal capacitor (called sample and hold, S&H capacitor) needs to be charged.
Charging the capacitor takes time. There is a time constant called tau. See here
https://en.wikipedia.org/wiki/RC_time_constant
For a 10-bit ADC you need at least 7 times tau to charge the S&H capacitor to within 1LSB.
The time constant depends on your source impedance e.g. the Potentiometer, the sampling capacitor is fixed. If the resistor value is too high the capacitor is not charged or discharge enough by the time the sampling is finished. The datasheet states the impedance of the source should be 10kOhm or less.
When you leave a pin open the impedance is many MOhms, so the S&H voltage stays near the level it was before.
If your potentiometer is larger than 10kOhm you could add some buffer capacitor at each analog pin.
For 10-bits I would suggest to make the capacitor more than 2048 times larger than the S&H. Because if you remove or add 1 S&H charge the voltage would stay within 1/2 LSB. But, doing that will reduce the sampling frequency. You have to calculate tau with your source and the large capacitor.
What are the resistor values of the joystick?