I redid your last test on a standard Arduino and this seems to confirm your measurements.
Above is from switching AREF from internal Vcc to the internal 1V1 bandgap reference and settling time is just above 5ms. This is significantly more than one ADC sample as the AtMega328 datasheet suggests.
From AtMega328 datasheet:
The first ADC conversion result after switching reference voltage source may
be inaccurate, and the user is advised to discard this result.
Max analog sampling frequency, when using the Arduino core (analogRead), is just above 8kHz - so we would in fact need to discard up towards 50 samples to be safe. As such, the Atmel quote above seems somewhat of an understatement.
A related thread on the "AVR freaks" site suggests that the internal bandgap reference has issues with sinking current. In the context above this would be relevant as the 100n Arduino Aref capacitor is at 5V prior to switching.
This significant settling delay would severly limit the usefullness of doing ADC sampling in a round-robin fashion with alternating internal AREF's.