You should never connect a voltage source to any pin of an unpowered Arduino other than Vcc/Vdd. Power all connected devices up and down at the same time.
There are a very few circumstances where it is difficult to avoid doing so. For input pins, put a 10K resistor in series, to limit the current into the unpowered Arduino.
I am supplying regulated 5V at the AREF pin to use it instead of the internal reference for voltage measurements.
Use that same regulated 5V to power the Arduino. Problem solved.