Regardless, is there are way to avoid the 1200us slowdown? I'd like to be able to read a byte anywhere in the file in under 500us.
An entire 512 byte block must be read from an SD. You will never achieve less than 500 us. It will take much longer for a large seek since blocks must be read from the FAT (file allocation table) to find the correct data block..
You can avoid the extra I/O for FAT blocks by using a contiguous file with SdFat and do raw reads.
How large is your file? If it is small enough you could use an SPI RAM, pito has posted several topics about an 8 MB module.