massive 'occasional' SD latencey - how to solve ?

(on the due) the application is writing ~400kBytes/s to SD cards
this is occuring as the data is being written into a large (8x8k) round robin buffer
(all happening async, clocked from PWM to a/d & dma'ed drirectly to the buffers)

most of the time it rolls along using at most 2 buffers but a couple times a min. it will stall & consume (overrun) 16 buffers worth ....

this is using sdfat lib (very nice BTW) as well as Utouch & UTFT (also very nice )
few oddities -- SPI_DIV6_SPEED is slower than SPI_QUARTER_SPEED though the docs make it sound faster ..

the alternates are to shrink the data & keep the same rate , 10bit into 8bit and or decimating the input ....

anyway / if anyone as an idea that doen't require me shrinking the data thoughput ...

oh & another question ... does any one sell CF shields ?? looks like they used to exist but have been taken off the market ...

It could be because of two reasons:-

  1. You hit a bad block in the SD's NAND flash memory and you have to rewrite. Flash memory - Wikipedia
  2. You run out of pre erased memory and the filing system has to go and erase some in order to write the new data. This can happen because the SD filling system is always trying to use all the memory address locations to do what is known as wear leveling. This prevents certain address locations being written to too often and thus wearing it out.

used a brand new sd card / had the same result

All SD cards have occasional long latencies. There are a number of reasons such as wear leveling, pre-erasing large physical flash block, and more.

The SD spec allows occasional delays of up to 250 ms for SD/SDHC cards. Typical cards have around 100 ms max delay.

There is very little RAM buffer in SD cards to keep the cost low. You must design your application with very large buffers to accommodate these delays. I either use a RTOS to run multiple threads or capture data in an ISR.

There are several examples here Google Code Archive - Long-term storage for Google Code Project Hosting.. See AnalogIsrLogger20121219.zip, FreeRTOSBeta20130107.zip, and ChibiOS20130112.zip for examples.

There is no easy solution to this problem.

SD cards only perform well with large contiguous writes so preallocating a file helps. See the AnalogIsrLogger for this method.

ok .. this seems to be a show stopper for what i need to do

i'm considering creating a shield (fpga/dynamic ram) that would be a giant fifo
8/16 word width with only a write/read/clear/empty/full , no address lines
..(this would not be random accessable)..