SD cards must be written in 512 byte blocks and random long write latencies are a prosperity of all SD cards. The SD spec allows occasional write delays of up to 250 milliseconds on a write but typical cards have occasional delays of more like 40-80 ms. A major reason for these delays is wear leveling of flash memory. Another reason is that flash erase groups are very large in SD cards which can cause the card to be busy.
To write fast, you need a large buffer and multiple threads or you can capture the data in an ISR and buffer it.
I have posted programs using both of these methods at http://code.google.com/p/beta-lib/downloads/list
AnalogIsrLogger20121219.zip uses an ISR to capture data.
ChibiOSBeta20121217.zip is a more general approach and uses ChibiOS/RT to run separate threads for data capture and writing to the SD. There are two data logger examples, one for Uno and one for Mega.
fastLoggerBeta20110802.zip has some additional examples of these methods but is somewhat out of date.