100+ encoders

No, just 3 pins if you daisy-chain. Then it becomes one slave with 256 bits of data to send. You don't need MOSI for input, and you don't need SS if it's the only slave on the bus. So just MISO, CLK & LATCH.

You would probably need to buffer the clock & latch lines, with such a large fan-out.

EDIT: actually, the '165 doesn't have an output enable pin, so it would have to the the only slave.