SD card timing questions.

Hi, I've been working on a project that involves processing data from lots of sensors and logging it to an SD card. The project has grown and grown over the months, and has got to the point where it's starting to grind to a standstill.

So, for the first time I've started investigating how long each bit takes. What I've discovered, is that about the single longest, most time consuming activity is writing to a file on an SD card. As listed here.....

Opening a file normally takes about 15ms but can take up to 50ms.
Closing a file takes about the same time as opening it.
Writing a single line to the file takes about 0-2ms for most lines, but every so often a write() will take 80ms. (I assume some sort of 'commit' is taking place at this moment?)

Here's the code I've used to demo this (it's very simple code, but it is a little verbose with all the timing information I've put in it)

    unsigned long A=0;
    unsigned long B=0;
    Serial.print("\n######## TEST ");Serial.print(testNum);Serial.println(" #########");
    Serial.println("Creating a random-name file and adding lots of lines to it.");
    SD.begin();
    int fileNo=random(0,999);          // Random file number between 0 and 999
    sprintf(fileNum,"%.3d",fileNo);    // Format the file number as a three digit string with leading zeros
    strcpy(fileName,"test");
    strcat(fileName,fileNum);
    strcat(fileName,".txt");
    Serial.print("Opening file <");Serial.print(fileName);Serial.println(">");
    A=millis();
    File f1 = SD.open(fileName, FILE_WRITE);
    //file1.open(fileName,0);            // Open the file for writing, if it doesn't exist, create it
    B=millis();
    Serial.print("Duration (open) <");Serial.print(B-A);Serial.println(">ms");

    // Add the first line....
    f1.write("Opening line for file named......");
    f1.println(fileName);

    for (int n=0; n<1000 ;n++)
    {
      //Add line to the file
      A=millis();
      f1.println("ABCDEFGHIJKLMNOPQRSTUVWXYZ blah de blah......");
      B=millis();
      Serial.print("Duration (line number <");Serial.print(n);Serial.print(">) <");Serial.print(B-A);Serial.println(">ms");
    }

    // Add the Last line....
    f1.write("Closing line for file named......");
    f1.println(fileName);

    Serial.println("Closing....");
    A=millis();
    f1.close();
    B=millis();
    Serial.print("Duration (close) <");Serial.print(B-A);Serial.println(">ms");

The problem is this, some of the data I'm trying to log arrives at unknown intervals via a serial port. If a 'burst' of data happens to arrive during the one of the 80ms pauses that occasionally happen as lines are being written, then there's a danger of the data arriving at and overflowing the serial port before there's any chance of reading it.

Questions...
Why does writing to a file take so long? Are these figures normal, or am I doing something wrong?
What's actually happening when a single write() suddenly takes 80ms?
Is there any way I can predict when one of these extra-long writes is about to happen?

All advice gratefully received!

Thanks

SD cards are block devices with 512 byte blocks. An entire block must be read/written so SD libraries have a 512 byte cache buffer.

SD cards are flash memory devices so blocks must be erased before they can be written. Flash memory only supports a limited number of erase/write cycles so the SD flash controller occasionally remaps blocks and copies data for wear leveling.

In short, you can't avoid some fairly long write times. The SD spec allows up to 250 ms for a write.

Opening a file normally takes about 15ms but can take up to 50ms.

FAT file systems do not sort or index files so open does a sequential search for the file. The time depends where the file's directory entry is located and can be much longer than 50 ms.

Closing a file takes about the same time as opening it.

Closing a file requires writing cashed data and reading, updating, and writing the directory entry so this also can take more than 250 ms.

Is there any way I can predict when one of these extra-long writes is about to happen?

No, it depends on the algorithms in the SD card flash controller.

It is possible to log data reliably at high rates. See this example Try this super fast analog pin logger - Storage - Arduino Forum

Thanks fat16Lib

So I guess the 80ms write() is actually the moment that the entire 512 byte block gets written?

Does this mean that if I record the number of bytes I've sent to a file, I can take a guess at when the next block is due to be written? Example, if I know I've written 500 bytes to a file, and I'm about to write another 25, then can I assume that the final write is going to take a long time?

Thanks for help - guess from your name that you know a fair bit about this subject!

Cheers

Does this mean that if I record the number of bytes I've sent to a file, I can take a guess at when the next block is due to be written?

No, a block write usually takes much less, 1-2 ms. the 80 ms is due to the SD flash controller doing house keeping. There are also writes to maintain the FAT and other file structures.

SD cards are designed for large contiguous writes. Arduino does not have sufficient RAM to optimize latency for file system writes.

The above link shows how to avoid these problems by doing raw write to a pre-allocated contiguous file.

Thanks for help - guess from your name that you know a fair bit about this subject!

I wrote SdFat. SD.h is a wrapper for a very old version of the SdFat library.

due to the SD flash controller doing house keeping

Is this 'flash controller' on the arduino, or built into the SD card itself? If it's on the card, how can I tell if I've got a good one or not? Is there anything I need to look for when buying an SD card? - The only metric I've ever seen written on an SD card is it's size, how can I tell if I've got a fast or efficient one?

Cheers

Is this 'flash controller' on the arduino, or built into the SD card itself?

The flash controller is built into the card and you can't get much info about it. The actions of the controller depend on the access history to the SD card and you can't know that since it depends on the implementation of software on all the computers you have used.

Buying a "high performance" card won't help because cards are designed for devices that have lots of RAM for buffering and use 4-bit SD mode.

The Arduino has little RAM and uses the old SPI mode so you won't get great performance from high end cards.

I have posted lots of programs that deal with this problem. They are either multi-threaded RTOS based or collect data in an interrupt routine. These programs use RAM to buffer data and separate data collection from writing to the SD.

Serial data is received in an interrupt routine so their is some buffering. I wrote a Serial logger with more flexible buffering.

This library GitHub - greiman/SerialPort: Enhanced Serial Library for Arduino has a high performance Serial logger.

Thanks fat16lib,

One more question please, the documentation says that I need to close() or flush() the file to ensure that data actually gets written to the. If I have failure of some sort (say a power loss) what's the most data I might loose? The most recent 512 byte block? or the data written since the flash controller on the SD card last did some house-keeping?

Thanks

The most recent 512 byte block? or the data written since the flash controller on the SD card last did some house-keeping?

You will lose all data since the last flush() or close(). This is when the file's directory entry is updated. You may also corrupt the file system since there may be a chain of lost allocation units.

You may want to consider using a RTOS. The nilSdLogger example for this small RTOS is very fast, solves the SD latency problem and can be modified easily.

Look at nilSdLogger in NilRTOS20130720.zip here Google Code Archive - Long-term storage for Google Code Project Hosting..

You will lose all data since the last flush() or close(). This is when the file's directory entry is updated.

Is there a limit to how much data I can write to a file before doing a close() or flush()?. Can I open a file and write (say) 10 Mbytes of data and then do a close(), or do I need to do flush() periodically?

I guess I need to code some automatic flushing to cope with situations where power might fail un-expectedly, otherwise I'm likely to loose data.