I did find that I'm not setting high speed mode :-[
This might improve matters.
Another thing you might consider is setting up a 'double buffer' system, where you're reading values into one buffer under interrupt control whilst writing the other. You can tune the sampling rate so that you get a consistent period.
Another thing to do might be to break the 'write sector' function so that you can write bytes at a time... You'd have to re-send a command every 512 bytes or so, but that shouldn't take long - there's a latency of about 12 bytes as the write is finished up, then the next sector is primed.
Sorry if that all sounds quite complicated - but it has to be. You don't get something for nothing on a microcontroller 