This is one way I see it working,
It would need the above for every 8 pins you wanted to handle, but if you stick to the standard shield (not Mega) that's only 3x the above I think.
Bear in mind that some control circuitry would be required as well.
XPS = crosspoint switch
BUF = buffer, probably TXB0108
Rob