I can't quite follow what you've done without a schematic, but maybe you should look at a MUX like the 74CBT3257, I've used these before to switch the SPI port on small AVRtiny chips for programming.
If you have a master CPU it can use one signal to both switch the MUX and signal the slave that it has access.
Rob