I'm currently thinking after browsing the internet for a solution is to use a DPDT switch IC to switch between the devices.
I wouldn't do that, there is an easier solution:If you have an Arduino that has multiple hardware serial ports, just use two of the available ports (one for each device). If you only have one hardware serial port, just use softserial.h to make two software serial ports and keep the hardware serial port for debugging.
Would there be any significant performance cuts or interference with features such as timers using this approach?
Get a display with SPI interface?