You can multiplex the physical part of the serial connection with CD4066 CMOS analog switches. They're bidirectional, as they are basically just a FET swtich.
Assuming you switch them on at the right time, you can move the serial port connection from one device to another.
D