Noise from Relay Channel Selector

tried out this setup for a channel selector that separates both the tip and the sleeve of an audio signal.
the problem is that I am getting noise that I can only remove by connecting all sleeves together.

how could I keep the signal noiseless, while still keeping the sleeves isolated?

The sleeves are ground. I don’t understand why you want to isolate it.

I separated ground as a measure to avoid ground loops between the outputs.

Not sure what’s getting connected to the two inputs. If it’s working to tie the ground together. I don’t get why not. Ground loops are created by additional paths to ground with different potentials. Not necessarily having equipment tied.

Do you mean switching noise - which is to be expected if you allow the grounds to float when not connected, or some other sort of noise?

If it is switching noise, then it is the ground loops that are causing it. Because there may be a voltage difference between the ground points, you are making and breaking the connection to that voltage, generating an impulse. Clearly in whichever circuit presently switched, there is a ground loop, so you have not avoided it anyway.

I think it’s causing more problems not tying the ground in this situation. If the two audio sources are not otherwise tied together you should have the ground tied in the output.

ran a test again an found out that there is no noise in the above configuration (one microphone input and two isolated speakers).

however, in reverse direction, that is one speaker as output and two separate inputs (mic + guitar) - there is noise being picked up. (unless I make a common ground)

I have to admit that at this point, the ground loops are merely theoretical and I will follow your advice to tie all grounds together. It's just a curiosity. Why is it quiet in one direction and noisy in the other?

(Paul__B. Switching noise is not an issue right now)

Common all the audio grounds - you are probably picking up capacitive mains hum from the
disconnected input device which is left floating.

Don't connect the audio grounds to anything else.