Assuming these 4 digital audio sources are all from independent consumer-level equipment, they will all be running at their own distinct sample rate. Even if they all use the same rate, like 44.1 or 48 kHz, they won't be exactly the same or consistently in phase with each other. Only very high-end pro audio gear does that, and only when you do something like run a "word clock" coax cable to every device.
If you will only pass the data from 1 input to the output, without any changes to the data, and without any effects like smoothly fading between inputs, maybe you could just make something that's like a physical switch, but controlled by a program.
But if you want to actually do something like mixing, smooth fading, automatic gain control to match levels, you'll need to actually receive the data from all the sources. You'll need to do some sort of resampling or timing adjustment to the data streams. This takes quite a lot of processing power to do with high quality results.
Today you probably need PC class hardware. On Linux, you probably need something like PulseAudio which isn't normally used on low-end boards like Raspberry Pi. The lighter systems like Jack might also be able to do this, but I'd be more skeptical of those. PulseAudio is probably your best path if you try to use a Linux board.
On microcontrollers, so far I'm not aware of anyone yet supporting multiple non-synchronous digital sources. I'm the author of the Teensy audio library, and someday I hope to be able to support this sort of thing. Recently I merged a code contribution which does this for the S/PDIF input on Teensy 4.0 & 4.1, though it looks like it is not yet working smoothly. Hopefully in time that will be fixed, and farther in the future hopefully we'll be able to support multiple I2S slave mode (where the external world controls the sample rate) and resample those all in real time. But today that sort of software support is more wishful thinking than reality. Most of the raw pieces exist, but with bugs and limitations and not yet integrated so they're easy to use.
Hopefully this long-winded message isn't too discouraging. I wish I had a better answer. But at least knowing about the need to resample or otherwise deal with the not-synchronized digital audio sample rates from consumer devices might help?