Yes that's too much of a compromise 
I've been experimenting with other flags and DMA settings and I can't get the lost speed back.
Also after modifying the bench example to output to the VGA library, it turns out that setting SPI_MR_WDRBT is slower than disabling DMA (#define USE_SAM3X_DMAC 0) when using both libraries together. There is no way I can find of getting the two libraries to share DMA without either SdFat going too slow or getting SPI underruns, or the VGA library getting visual glitches. Disabling DMA in SdFat is faster by about 50% (800KB/s v 1200KB/s) with no glitching.