I'm building a MIDI controller. If the Arduino is connected to the MIDI OUT while you are updating the program, it causes glitching and spurious noises.
And typing this answer out has actually given me the answer to my own question! I think I have to put the switch in-line with the MIDI out, not the USB. Doh! Should be easy now.