Can't see any problem with it so far. I have been quite deep into proAudio hardware and this kind of circuit is very common in old-school digital architectures. The pull-up resistors at the CS inputs of the displays are of course essential because when a display is inactive the multiplexer port will be high impedance.
However, I am not sure what max. speed could be handled by that circuit. I wasn't able to measure the actual SPI clock yet but it should be more or equal 8 MHz.
[Edit:] The TI datasheet of the 74HC4051 states a frequency response of the multiplexer switches of > 40MHz. Should be good for any SPI CS application with DUE.
BTW: More than 8 displays will simlpy require to expand the multiplexer size to 16, 24, 32 ..... This is fairly easy. I am just concerned about speed.