Fast 16bit ADC using Arduino Due and AD977

The SPI.transfer() function library description says that the "(10,0x00)" parameters start the byte transfer by pulling pin 10 low from its normal high state. Using "(10,0xFF)" would not do this and would mess up the SPI timing. Therefore the invertor is necessary.

Both versions do pull SS low but the first version sends a null byte to the chip while the later sends only one bits. SPI is capable (it even enforces) of sending and receiving concurrently, so if you want to receive a byte, you have to also send one. Usually a null byte is sent while you expect a receiving byte.

I understand that baud rates are often only approximations to the specified SPI rates

You mix things up. The baud rates you specify are for the UART communication and not SPI. You don't communicate with the PC by SPI.