If I understand what you question is correctly, then it should be no problem.
My advice would to be to go with a simple multiplexer such as the 74CD4051. It uses 3 digital select pins and then has 8 analog channels.
Its just your basic multiplexer. It is capable of demultiplexing as well, although I think most muxes are capable of demuxing but don't quote me on that.
You may run into an issue, say the person on one end is talking and the other interrupts the first, then you'd be sending data through a single channel. I have never tried that, although I suspect you would have some sort of issue. A possible fix is to just 2 muxes and have one for listening and another for talking. I can make a diagram if you need if that doesn't make sense.