I still think it's something related to grounding somehow.
If I remove the MIDI code and put in a Serial.begin(), then send stuff from the serial monitor in the IDE, no noise.
GRRRR... I can unplug the arduino entirely except for the MIDI cable and the audio output cable. And I still get the noise, but quieter. Something must be wrong with the breakout?