(Thinking out loud, disclaimer, never did this myself)
You could cross-breed SPI and I2C. I2C allows addressing up to 100+ slaves.
-
set one slave as the active one. The Arduino sets its own CS pin HIGH after the appropriate I2C request.
-
send 'big' data with high speed over SPI back and forth
-
deselect slave. The Arduino sets its own CS pin LOW after the appropriate I2C request (or time out !)
-
go to 1

should work, but it depends on your timing requirements