I had an idea a while back where you use 2 4051 mux IC's to multiplex the RX and TX to 8 devices.
One 4051 would handle all the RX lines the other all the TX lines.
In theory this should work using only a few Arduino pins. And it could be extended to as many as needed.
The problem with this and all other multiplexing / switching solutions is that it will only work in a system where you only have "call / response" type communication. You can only deal with one line at at time, so if device y is sending data to you while you are reading or sending data to device x, the data from device y will be lost.
But for controlling a bunch of displays it would probably work well.