Explanation of SD write() and flush() please?

Hi,

Academic question only....

First I'll start with what I think I know about using SD cards, (based on experimentation and what I've picked up on this forum)

Each time I have 512 bytes of data in my buffer I do a write() - this takes typically 7-8ms, effectively this copies my data into the SD cards own buffer.
This does what I think of as a 'soft write' - it's on the disk, but if I loose power before a close() then the data's lost.
Every now and again I have to do a flush() - this is like a 'commit' in database parlance, once the flush() is done, then even if I loose power then the data should still be in the file. the flush() also seems to take about 8-10ms.
A close() seems to the be the same as a flush, except that after a close() no further writes() can be done until it's re-opened.
Right so far?

So...
Can I write() a buffer of more than 512 bytes to the SD cards in one go?
Once I've done a write() (and waited 8ms), why does the flush() also take nearly as long? the data is already on the card isn't it?
How often do I need to do a flush()? if I'm not worried about battery failure/data loss, can I write() a gigabyte of data (in 512 byte blocks) and then do one big flush() at the end?

Right so far?

Yes.

Can I write() a buffer of more than 512 bytes to the SD cards in one go?

No. The typical block size is 512 bytes, so that is the standard buffer size. If you are using your own 512 buffer (complete unnecessary), then you've already used 1/2 the memory of a 328-based Arduino.

Once I've done a write() (and waited 8ms), why does the flush() also take nearly as long? the data is already on the card isn't it?

If your call to write() fills the SD library's buffer, then the buffer is copied to the file. Otherwise, write is VERY quick.

The call to flush() does more than just write to the file. Look at the source code to see exactly what it does. (Though I'm not sure that anyone by fat16lib understands the code...8))

How often do I need to do a flush()?

Lets ask, instead, how much data you can afford to lose. How much would that be? When you've written that amount, flush() makes sure you don't lose that data.

can I write() a gigabyte of data (in 512 byte blocks) and then do one big flush() at the end?

All flush()s are the same size.

No. The typical block size is 512 bytes, so that is the standard buffer size. If you are using your own 512 buffer (complete unnecessary), then you've already used 1/2 the memory of a 328-based Arduino.

Whys is this 'completely unnecessary'? the tests I've done (using the SD library, not any of my own code) indicate that a write() takes about 6-8ms irrespective of the amount of data being written, if I do ten separate writes() each of a 50 character record then that takes 10*8ms i.e. 80ms, but if I save up all my data into one big 500 character buffer and then do one write() it still takes 6-8ms. So if performance is an issue and the memory is available then isn't it better to use your own buffer? Maybe I've misunderstood (quite likely in fact!) but several of the posts I've read seem to suggest I should be using as big a buffer as possible?

flush() does the same thing as close() except for marking the file closed. This requires writing many file structures, not just the cached data.

You don't need to call flush unless there is a chance of your program crashing before you close the file.

Here is a simple program that writes 524,288 byte (1024 blocks on the SD).

It uses buffer sizes of 1, 2, 4, 8, 16, 32, 64, 128, 256, and 512 bytes.

#define SD_CS_PIN SS
#define TEST_FILE_NAME "testfile.txt"
// 1024 block file
#define FILE_SIZE 1024UL*512UL
#include <SPI.h>

// use next line to test SD.h
//#include <SD.h>

// use next two lines to test SdFat
#include "SdFat.h"
SdFat SD;

uint8_t buf[512];
File myFile;

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);

  // Wait for USB Serial
  while (!Serial) {

  }
  if (!SD.begin(SD_CS_PIN)) {
    Serial.println("SD.begin failed!");
    return;
  }
  for (size_t n = 1; n <= 512; n *= 2) {
    SD.remove(TEST_FILE_NAME);
    myFile = SD.open(TEST_FILE_NAME, FILE_WRITE);
    if (!myFile) {
      Serial.println("open failed");
      return;
    }

    // Write file data
    uint32_t us = micros();
    for (uint32_t i = 0; i < FILE_SIZE; i += n) {
      if (n != myFile.write(buf, n)) {
        Serial.println("Write failed");
        return;
      }
    }
    us = micros() - us;
    myFile.close();

    Serial.print("buffer size (bytes): ");
    Serial.print(n);
    Serial.print(", time (sec): ");
    Serial.print(0.000001*us);
    Serial.print(", rate (KB/sec): ");
    Serial.println(FILE_SIZE / (0.001 * us));
  }
}

void loop() {
  // nothing happens after setup
}

Here are the results of running it with the SD.h library. SD.h uses a very old version of SdFat at slow SPI speed.

buffer size (bytes): 1, time (sec): 22.21, rate (KB/sec): 23.61
buffer size (bytes): 2, time (sec): 12.78, rate (KB/sec): 41.01
buffer size (bytes): 4, time (sec): 8.21, rate (KB/sec): 63.82
buffer size (bytes): 8, time (sec): 5.79, rate (KB/sec): 90.60
buffer size (bytes): 16, time (sec): 4.61, rate (KB/sec): 113.76
buffer size (bytes): 32, time (sec): 4.05, rate (KB/sec): 129.52
buffer size (bytes): 64, time (sec): 3.73, rate (KB/sec): 140.38
buffer size (bytes): 128, time (sec): 3.61, rate (KB/sec): 145.22
buffer size (bytes): 256, time (sec): 3.52, rate (KB/sec): 149.12
buffer size (bytes): 512, time (sec): 3.08, rate (KB/sec): 170.24

Here is the result with a new version of SdFat.

buffer size (bytes): 1, time (sec): 18.26, rate (KB/sec): 28.71
buffer size (bytes): 2, time (sec): 9.75, rate (KB/sec): 53.76
buffer size (bytes): 4, time (sec): 5.46, rate (KB/sec): 96.02
buffer size (bytes): 8, time (sec): 3.33, rate (KB/sec): 157.32
buffer size (bytes): 16, time (sec): 2.27, rate (KB/sec): 230.65
buffer size (bytes): 32, time (sec): 1.76, rate (KB/sec): 297.35
buffer size (bytes): 64, time (sec): 1.66, rate (KB/sec): 315.02
buffer size (bytes): 128, time (sec): 1.72, rate (KB/sec): 304.76
buffer size (bytes): 256, time (sec): 1.69, rate (KB/sec): 309.48
buffer size (bytes): 512, time (sec): 1.59, rate (KB/sec): 329.91

There is little gain in speed after a buffer size of 32 bytes. For a one bytes buffer, the rate is 28.71 KB/sec so the time to write a single byte is about 0.035 ms.

For a 32 byte buffer the rate is 297.35 KB/sec or 0.0034 ms per byte.

Note the slight increase in speed for 512 byte writes. SdFat does not cache these writes since they can be directly written to the SD. This is only possible when writing a full SD block and the file position is on a block boundary.

Ok, I've just gone back to my test routines and have added some functionality to capture and average the write() times. In one of my replies I stated...

a write() takes about 6-8ms irrespective of the amount of data being written

Which would make it sensible to buffer-up a big block of data and then do one big write.

I've re-done my tests and added a routine to store and average all the write() times for each record, and there is a noticeable increase in duration as the amount of data being written goes up.

So basically there's no point in me doing what I was doing and buffering my data into 512 character blocks before doing a write();

One last question (I hope), I store my records as an array of chars 'lineOfData', and I write to the SD card like this.....

f1.write(lineOfData, strlen(lineOfData));

but purely by accident I discovered this seems to work just as well....

f1.write(lineOfData);

They both work, is one method supposed to be better than the other?

Whys is this 'completely unnecessary'?

Putting data in your buffer, and then handing the full buffer to the SD class to copy to its buffer, filling it, so it can write it's buffer to the file doesn't strike you as unnecessary?

Picture it like this. The SD class is going to pour water from its 5 gallon bucket down the drain (into the file). You can give the SD class water one cup, one drop, one gallon, or 5 gallons at a time. Regardless of how much water you give it, the water you give it is going into its bucket first. When its bucket is full, it will be emptied down the drain.

So, does it make sense for you to have a 5 gallon bucket AND for the SD class to have a 5 gallon bucket, when you are putting water in your bucket one drop at a time?

Couldn't you, just as easily, kick your bucket out of the way, and give the water to the SD class one drop at a time?

The buckets are expensive. Don't buy two of them when one is sufficient.

Thanks PaulS

you're right of course, I had though a buffer was a good idea because I incorrectly thought that ever write() takes the same duration irrespective of number of characters being written, in which case reducing the number of writes by doing one big write would have saved time overall.

Thanks