I2C requires two pins: SDA and SCL. They are shared between all the displays. You will have to set them all to a different bus address (the I2C backpacks have solder jumpers for this), but then it's just two wires for all four displays.
An SPI backpack needs 7 pins total as mentioned above.
The SPI bus (MISO, MOSI and CLK) is hard wired to a specific set of pins, it's built into the MCU. You can use any digital I/O for the four CS lines.