Analog measurements are relative to whatever you are using for the ADC reference voltage.
It is not clear from your post what you are doing, but from the symptoms, the ADC reference voltage must be changing as you switch power sources.
Keep in mind that it is quite easy to destroy the ADC, and possibly also the entire MCU if you do not follow all the rules in the data sheet for how you connect and programmatically set up the ADC reference.
For informed help, post the code, using code tags, and a wiring diagram (not a Fritzing mess).