I will have to learn about shift registers.
I also think I can loose the SPI display and substitute an I2C 20x4 line display.
I will do a re-draw of the circuit and see...
Well now the original question is no longer, since I will not be using SPI.
Can an output pin be changed on the fly to input?