Unfortunately you have chosen the wrong parts for your project.
An absolute A/D (the ADS1115) combined with a ratiometric sensor (the pot) makes results highly dependent on the power supply. Same goes for the A/D of the ESP.
You can fix this by also measuring the pot supply with the ADS1115 (A1),
and using this as a factor to compensate pot results (complicated).
Or ditch the ADS1115 and use a ratiometric A/D chip (one without builtin reference).
Leo..