Class "Log_File": SD card writing with "cache" in order to reduce wear level

To my knowledge, no matter how few bytes you want to store on a SD card, always an entire sector is (over-) written, usually 512 bytes. This usually should not matter much, but when you write many small chunks of data very often to the card (--> data logger) the picture may look different.
So in order to decrease the wearing out of the SD card in a datalogger application, I wrote this class "Log_File" - the downside of course being that the buffer needs RAM.

Data that should be written to the card is instead only written to a buffer.
If the buffer exceeds a certain length, the information is stored on the card and the buffer cleared.

class Log_File{
  //String filename;
  //int block_size;
  public:
    Log_File(char filename[13], int block_size);
    Log_File(char filename[13]);
    void log_data(String mydata);
    void store_data_now(String mydata);
    void store_data_now(void);
  private:
    char * _filename;
    String _buffer_string;
    int _block_size;
};

Log_File::Log_File(char filename[13])
{ _filename = filename;
  _block_size = 512;
  _buffer_string = "";
}
Log_File::Log_File(char filename[13], int block_size)
{
  _filename = filename;
  _block_size = block_size;
  _buffer_string = "";
}


void Log_File::store_data_now(void)
{  //store buffer immediately
    File dataFile = SD.open(_filename, FILE_WRITE);
    dataFile.print(_buffer_string);
    dataFile.close();
    _buffer_string = "";
}
void Log_File::store_data_now(String mydata)
{  //store buffer immediately
    _buffer_string += mydata;
    store_data_now();
}
void Log_File::log_data(String mydata)
{  //write mydata to buffer, if buffer exceeds size, write buffer to SD card
    _buffer_string += mydata;
    if (_buffer_string.length() >= _block_size){
      store_data_now();
    }
}

How to use it:
Log_File my_new_logfile("name.dat", 512); //Name of Logfile and buffer size

my_new_logfile.log_data("abc"); //writes "abc" to buffer
my_new_logfile.store_data_now("def"); //adds "def" to buffer and writes buffer to card
my_new_logfile.store_data_now(); //writes buffer to card

This was one of my first experience with classes. So if someone could please verify the following:

Log_File my_new_logfile("name.dat"); //if no buffer size is given, the standard value of 512 should be used automatically

Any comments or suggestions for improvement?

Any comments or suggestions for improvement?

The SD and SdFat libraries already use a buffer. What advantage does your class offer?

How does pissing away resources on the String class help?

The SD and SdFat libraries already use a buffer. What advantage does your class offer?

Indeed, the SD library uses a buffer but the way I understand the code it is used to break down a big amount of data into smaller chunks and store those on the card. Yet all of these parts are written consecutively without any break in between.

I come from the opposite problem: Only a few bytes to store. So it seemed reasonably to my to first collect a lot of those bytes and then store them at once. That's why I created the above code to reduce actual writing incidents.

How does pissing away resources on the String class help?

I am fully aware of the sacrifice of RAM. That's why I allowed for a variable buffer size.

That's why I allowed for a variable buffer size.

You should malloc that buffer ONCE, not let the String class thrash all of your memory.

Your use of the buffer is not contributing anything over the standard buffer that the SD (and SdFat) class provides. The SD card is written to in 512 byte chunks. Your call to dataFile.print() in Log_File::store_data_now() writes to the SD class' buffer. Closing the file commits the SD buffer to the card. That, or the buffer being full, are the only times that the SD class actually writes to the file.

Your class is, therefore, double buffering data, in a resource-wasting way, which in no way benefits the card.

In order to verify your statements about the dual used within the normal SD library use I just ran a test sketch like this (so no of my library included whatsoever)

File datafile = SD.open("datafile.txt", FILEWRITE);
while(1){
  datafile.print("data");
}

So just the standard SD library and continuously writing data to the card, but without the close() command.

When I removed the SD card and looked at it with the computer I found the file, but empty. This contradicts your statement

That, or the buffer being full, are the only times that the SD class actually writes to the file.

doesn't it?

This contradicts your statement doesn't it?

No. Removing the SD card while there is a file open is a great way to corrupt the file. Which is what you did.

Prove, or disprove, my "claim" by writing 80 bytes to the file (actually buffer) 20 times, and then close the file. Time how long each write takes. 3 of them will take significantly longer than the others, where the buffer full of data is actually written to the card. The other 17 calls, that just write to the buffer, take next to no time to execute.

Due to some work related issues it took me some time to verify PaulS statements. He was right in all points.

Using a logic analyzer I could confirm that the standard SD library
a) buffers data before writing it on the SD card until the 512 bytes of the buffer are full
b) only if the buffer is full or the "close" command is issued, the data in the buffer is written to the card and the buffer cleared
c) storing the 512 bytes of the buffer on the card takes about 1.7 ms (including overhead, so ~300kbyte/s)

So, frankly, writing my library was a waste of time. Please discard it.

One question remains though:
512 bytes for the buffer seem a lot to me. Is it necessary / possible to reduce the size, if the program itself already needs lots of RAM?

halfdome:
b) only if the buffer is full or the “close” command is issued, the data in the buffer is written to the card and the buffer cleared
c) storing the 512 bytes of the buffer on the card takes about 1.7 ms (including overhead, so ~300kbyte/s)

You can also call flush (SD) or sync (SdFat) to update the FAT occasionally so that your file is saved in the event of a power loss or program crash. You could close and reopen to achieve this but flush/sync is more efficient.

So, frankly, writing my library was a waste of time.

We already knew that. :slight_smile:

I was originally confused about how SdFat/SD updated the card as well. In fact I made a couple of minor modifications to the write() method in SdFat, thinking I was fixing something (that didn’t need any fixing).

One question remains though:
512 bytes for the buffer seem a lot to me. Is it necessary / possible to reduce the size, if the program itself already needs lots of RAM?

I believe 512 is the smallest block size. There are other tricks to reducing RAM requirements but they involve loss of flexibility. For example, Petit FAT is an implementation that uses a lot less RAM but requires one to pre-allocate a limited number of files of fixed length.

A simple way to save a little bit of ram is to use SdFat instead of SD. That will save you about 65 bytes up front. It also appears to use about 65 bytes less stack space. So you gain about 130 bytes by switching over to SdFat. There’s really no good reason to use SD that I can see.