No, that's how it's done. A common mantra around here is "tie ALL grounds together" because everything needs to know what +5v is in relation to (for example.) Same goes for audio.
In your multimeter example, you're grounding the negative probe. Try measuring the voltage at the RCA output with just the red probe, leaving the black probe lying about on your workbench. That's what you get in your circuit if you don't ground the RCA terminal. So, that's why it's always done that way. :-)