If you're using the SPI pins for SPI then you should not use those pins also for something else. It sounds like you might be better off just using a Mega, which should provide enough memory and pins for everything. Of course that's not the only option. You can add more pins to your Nano via external components. You might be able to work within the available memory by modifying your code.