As audio is AC voltage your switching device choice is best met with either an analog switch chip (CD4066BM/CD4066BC ?) or a simple relay. As the audio AC signal has both a positive and negitive portion any analog switch device will require both positive and negitive supply rails. This can be a little tricker then it may first appear and can lead to hum being injected onto the audio signal. Your idea of somehow just switching grounds on and off is certainly not sound (pun intended).
All in all I'm in favor of simple 5vdc relays for such audio switching applications. While it's old tech, it's also pretty fool proof and easy to get working. Here is a simple relay that can be powered directly with an arduino output pin as it only draws 28ma of current, not much more then an LED would. One should however wire a reverse biased diode across the relay coil terminals for transient protection. Being it has DPDT contacts, the one relay can switch both the left and right audio signals at the same time with the one relay.http://www.allelectronics.com/make-a-store/item/RLY-538/5-VDC-DPDT-SUB-MINI-RELAY/1.html