But that breakout does have an optoisolated input, so shouldn't it be fine?
MIDI inputs are always opto isolated, it is part of the spec.
The grounding issue must be with the MIDI connection;
No it is as you point out optically isolated so there can't be a grounding problem.
it's fine as long as there's no MIDI input.
Ring any bells?
I even tried connecting a battery pack between the pins of the connector (obviously with the midi cable disconnected) and I get noise.
When you do that you send a junk byte into the serial input.