Yep, something is wrong when you turn off the ADC.
Two things I can think of. The first is that optiboot does not disable the UART after it is done trying to detect avrdude it just passes control to the user program. Second, the TWI (or I2C) pins have a weak pull up if that was enabled.
I would try Hans (MCUdude) repositories.
MightyCore turns off the UART (like the default Uno core), but I did not look in the core you are using (it probably does but you should check).