I would prefer to keep the audio signals isolated from the data board so would prefer not to connect the audio ground to the data board's ground and then switch the signal pins using transistors but would this be best? What do people think..... This is effectively like a computerised source switch for an amplifier.
I would use DPDT relays. (If you're switching unbalanced audio, or a 4PDT relay if you're switching balanced audio). The relay will be switching both the signal and the audio GND lines, and in no way will connect to your digital circuit's GND.
The relays will be controlled via logic level mosfets/or transistors, which turns on/off the individual relays.
You'll need a LOT of relays to do this, and increases as you have more inputs and outputs.
a 2x2 matrix, you'll need 4 relays (and 4 control signals)
a 2x3 matrix, you'll need 6 relays (and 6 control signals).
a 3x3 matrix, you'll need 9 relays (and 9 control signals).
You'll run out of AVR pins, so you'd probably want to use some external logic ICs to save some pins)
EDIT: This is just a diagram for one channel. So for stereo, you'd need another matrix board.
The "black lines" in this diagram represents both the audio signal and ground wire. X represents the DPDT relays.
Using this scheme, you can send IN1 to any combination or all of OUTA, OUTB, OUTC.