Diodes would work, but the thing that's bugging me is that the grounds of the two circuits are connected, and 12V and 5V must both be connected to ground somewhere in their circuits with pullups like this. So why should it matter where the common ground point is?
I guess maybe the issue is that GND connection might not even be there if the switch is off.
So is it okay as long as the switch is on, but not okay when it's off?