The mux is powered by the same power supply
Maybe this is my mistake. The 5Vcc is attached to Vin.
I obviously made a mistake with this. I've always worked with ATmega328s powered by a regulated 5V DC, as well as Arduino UNOs and ESP8266s, in the same way. I mistakenly didn't read the Nano's specifications, and I think that might be the source of the instability and temperature issue. Thank you very much for pointing this out.