Go Down

Topic: File.open on every loop() or only once in setup()? (Read 4107 times) previous topic - next topic

__Tango

I've got a sketch that's doing a lot of work and logging data to an SD card (once every 100ms).  I'm using the stock arduino SD library in 1.0.6 for now (soon to be 1.6.x).   Eventually, I may move to SdFat, but not at the moment.

Is it better to call SD.open, File.write, SD.close in the loop like this:
Code: [Select]

#include <Arduino.h>
#include <SD.h>

File dataFile;
unsigned long lastWriteTime = 0;

void setup() {
}

void loop() {
    if (millis() - lastWriteTime >= 100) {
        dataFile = SD.open("DataFile.txt");
        lastWriteTime = millis();
        // write data to file
        dataFile.write("blah");
        dataFile.close();
    }
}

Or call SD.open() in setup, hang onto the file object and then do File.write, File.flush() in the loop like this:

Code: [Select]
#include <Arduino.h>
#include <SD.h>

File dataFile;
unsigned long lastWriteTime = 0;

void setup() {
    dataFile = SD.open("DataFile.txt");
}

void loop() {
    if (millis() - lastWriteTime >= 100) {
        lastWriteTime = millis();
        // write data to file
        dataFile.write("blah");
        dataFile.flush();
    }
}


Why?

Is the answer different for using SdFat?

Is the answer different if I'm trying to write to two separate files at the same time (so open two files in setup or loop?)

Thanks!

fat16lib

#1
Jan 27, 2016, 11:30 pm Last Edit: Jan 27, 2016, 11:34 pm by fat16lib
Quote
Is the answer different for using SdFat?
No, the problem is the same for old SdFat and new SdFat.

The nature of SD cards and the FAT file system sets limits.  Also SdFat on AVR boards has a single 512 byte cache.  On ARM, the new SdFat has two 512 byte cache blocks so many fewer I/O operations are required.

Here is what happens with one cache block.

If you open a file on a FAT volume, a linear search of the directory is performed to find the file.  There are 16 directory entries per block so this search will require reading at least one block and many more blocks if there are lots of files in the directory.

After the directory entry is found, a seek to the end of the file is required.  This means reading one or more file allocation table entries to get the allocation chain.  Next the last block must be read, updated with the new data and written back to the SD.

Finally you close the file so the block with the directory entry must be read, updated and written back to the SD.

If you use write/flush the following happens.  The write call causes the last block of the file to be read, updated and written back to the SD.  The flush causes the directory block to be read, updated and written.

If you can be sure your program won't crash and you can close the file properly then just writing the file causes a single write for every 512 bytes.

On ARM with the new SdFat, the write/flush requires just two writes.  The last data block is in one cache and the directory entry is in a second cache.  Write updates the data block and flush updates the directory entry and writes both blocks to the SD.

If you have two files open it is still far more efficient to use write/flush.  I can't say more since I don't know the pattern of writes to the files.

Logging every 100 ms is marginal.  SD cards occasionally have very long write latency times.  This occurs mostly because of wear-leveling.  The SD controller may copy a very large, 128KB or more, block of data from one physical area of flash to another area.  The SD spec allows up to 250 ms for this operation.

If you need a high quality time series, time jitter must be a fraction of a percent of the interval between points or a fraction of a ms in your case.  Think of plotting the data when the accuracy of time for a point is poor compared to the accuracy of the point's value.

__Tango

Interesting.  Thanks for the in-depth explanation.  Maybe I can cache data and do write larger blocks less often (1000ms?). 

Seems like there is a slight efficiency gain in doing the open once write/flush method, especially as more entries are created within a directory, but it's not nearly as significant as I thought.  Is that correct? 

Thanks.

Go Up