The underlying assumption that because there are 4 signal lines and 1 ground line it would be easier to switch/disconnect the ground line is flawed.
The 'signal' really consists of the difference between signal and ground voltage for each of the 4 servos. Those 4 signals happen to share a common ground but they are 4 signals. If you disconnect the ground from the signal source but leave all the servo grounds connected together then you still have differences between signal and signal at the receiving end. You could disconnect each ground separately, but this is still switching 4 wires...