connect common ground via power switch

When using two separate power supplies (for servos and logic), I understand that there must be a common ground connection between both circuits for the servos to function properly. In doing so, I have seen dozens of tutorials where people use two separate switches to enable/disable power to each circuit separately. Wouldn't it be possible to use a single SPST switch to (dis)connect the common ground from each circuit? See ASCII art below for diagram. This seems like it would work, and eliminate a switch from the design. Are there other issues with doing this that I may be missing?

????    GND   ????
?9v????????????6v?
????  ??????  ????
 ?+   ?SPST?    ?+
 ?    ??????    ?
 ?   GND?????????
??????????   ???????
?ARDUINO ?   ?SERVO?
??????????   ???????
        ?????????
         SIGNAL

Nope, wouldn't work.

With the ground "closed", you get 2 circuits - 1 at 9v and 1 at 6v. With the circuit open you get one circuit at 3v. Best case - flat batteries. Worst case, reverse voltages in half the circuit kill your hardware.

Thanks majenko. My original plan was to use a single DPST switch to (dis)connect the separate positive connections. I guess I'll go back to that.

Or use one supply to switch the other supply on/off - either mechanically with a relay, or with something more solid state like a p-channel mosfet.