Because SPI transmits at the same time as it receives, it is always a challenge to be 1 byte ahead as a slave. That is, when you get an interrupt telling you an SPI byte was received (as a slave), you had better already have written the byte you want to send out. Not a big deal, but easy to get wrong.
Beware of slaves that are SPI-compatible but in fact are always driving the MISO lines (e.g., I've seen a WizNet chip do this).
I don't think "easy" will be a problem but "fast" entirely depends on the details.
If I2C is too slow then SPI is the next logical choice I suppose.