Why does AREF seem to pull up its input voltage?

You can look at the full schematic in the link if you like, but I think the fragment is pretty self explanatory. AREF is pulled up to +5V through the 5.1K resistor R6. Any attempt to use a different external voltage reference is going to be messed about by that resistor.