However, in high-speed mode (3.4Mbps), one cycle is only ~5us, giving a max sample rate of 212kHz(!) Is this to fast for the arduino?
Remember that the I2C bus is dependent on the clock rate you send it. We are just talking about the maximum rates here. There is no problem with sending data at any rate up to the maximum. With the arduino running at 16MHz I think the maximum clock rate for the I2C bus will be 4MHz but you would have to check the data sheet.
do I need a buffer on the SD card?
What do you mean by this? You defiantly need some logic level converters resistive potential dividers don't always work.
Current requirements for SD cards vary so you are better off regulating down to 3V3 from 5V.