fat16lib vs SD standard library

Hi all,
Been playing with a couple different libraries for working with SD cards, specifically William Greiman's Fat16lib and the 'SD' that comes with Arduino IDE as of 0022. Writing to SD cards is just plain useful, but my specific aim is to find / specify a lib that works well for a low-power Arduino variant I'm working on. ("Low power friendly" = less CPU cycles per bytes read/written, ability to reinitialize the card after cutting its power.)

'SD' scores usability points for being included right in the official Arduino distribution and examples. Unfortunately, it loses them (and then some) for having pin numbers hardcoded in the library(!)(a series of #ifdefs guesses the board you are using based on the CPU), meaning a user has to basically hack the library to use it with with something other than a commercially produced SD shield / well-known board. It gains some back for supporting FAT32 and SDHC cards, which to my knowledge fat16lib doesn't support.

Another ding for 'SD' is that I could only get it to initialize the card once. Subsequent begin() calls (e.g. after the card's power has been cut for a bit) fail. I haven't done enough digging yet to find why this is or how easy it would be to fix. Some of the cards I tested consumed as much as 20mA when idle, so it is important :slight_smile:

The real eye-popper was benchmarking them side by side on the same card (SanDisk branded 1GB microSD, FAT16 format). Replicating fat16lib's read/write benchmark as best as possible on SD, the following results are obtained on my board. Note that the test board runs at only 4MHz, so this should not be taken as an authoritative Arduino+SD benchmark of any kind. Both are using hardware SPI.

FAT16lib on SanDisk 1G uSD:
Free RAM: 3062
File size 1 MB
Starting write test. Please wait up to three minutes
Write 67.20 KB/sec

Starting read test. Please wait up to a minute
Read 86.57 KB/sec
Done

Arduino SD lib on SanDisk 1G uSD:

Initializing SD card...
initialization done.
Writing to test.txt, press a key to start...
File size
1
MB
Starting write test. Please wait up to three minutes
Write
3.36
KB/sec

Starting read test. Please wait up to a minute
Read
7.27
KB/sec
Done

Are these results consistent with expectations? That's a pretty big difference, and I have to say fat16lib's scores are impressive, almost enough to make me nervous (i.e. look for bugs). A slower read speed from 'SD' is forgivable considering it does not support reading entire buffers, only single bytes at a time, so the read test is not a 100% apples-to-apples comparison.

Besides all of what's above, are there any other reasons (bugs, etc.) to choose one over the other? Are there any of the less well-known SD libraries I should have a look at too?

fat16bench.pde (2.08 KB)

new_SD_bench.pde (3.08 KB)

The "standard SD library" is a wrapper for my SdFat library. It is slow due to choices made in the wrapper. It can be faster than my Fat16 library.

Fat16 was designed to be small so it only supports FAT16 file systems and gives up some performance.

I have completed a major rewrite of SdFat. The beta is here Google Code Archive - Long-term storage for Google Code Project Hosting.

Here are the results of running the bench.pde example on a 2GB SanDisk card:

Free RAM: 1075
Type is FAT16
File size 5MB
Starting write test. Please wait up to a minute
Write 194.49 KB/sec

Starting read test. Please wait up to a minute
Read 297.32 KB/sec
Done

This is an old blue card, not a "high speed" card. Often older cards are faster than the newer cards since the performance of the SPI controller is better. New high speed cards are not optimized for SPI transfers.

Here are the results on an old 512MB Corsair card which is no longer in production:

Type is FAT16
File size 5MB
Starting write test. Please wait up to a minute
Write 301.08 KB/sec

Starting read test. Please wait up to a minute
Read 327.44 KB/sec

Cool! I'll definitely give the SdFat beta a look!

Wow, the timing of this thread is amazing. I was just going to look around to see if there is any info on write speed for SD cards, as I am having a problem. I am sucking in binary data through a serial port at a pokey 19,200 baud, doing some processing, and then writing binary and text versions of the data to an SD card.

The writes seem to be holding me up enough that I miss a byte here and there, even though I am using an ATMega644p and one of its two USARTs. I am also using a data logging shield from ADAFruit and the Fat16 library. The card is a 2GB card from ADAFruit from "Transcend." Its actually a micro card in a regular size adapter. The IDE is Arduino-022.

I just downloaded the SDFat beta you linked here and read the docs. Very impressive! Its great that you not only took the time to write such a nice library but also prepared such nice docs, thank you!

I am having an issue. I can not get any of the sample apps to compile. The compiler complains of an error on line 40 of ArduinoStream.h: expected ')' before '&' token. I took a look and the reference parameter seemed to be declared correctly.

I tried several of the example files and each one gave me the same error. I'll try the SD classes without the stream IO and see if I have better luck.

Thanks!

Hm, well it all compiles fine on my Uno.

Using the bench example in the SDFat library, my Transcend card writes at 62 KB/sec. I threw a 256MB SanDisk card in and there was a slight difference. It wrote at 234 KB/sec!!!

I guess I'll try my app wit this SanDisk card.

Still, I would like to transition to this SDFat library, I just have to figure out why the code won't compile when "Sanguino" is selected as the board type.

Sorry SdFat no longer compiles for the Sanguino. The official software for the Sanguino is so out of date, 0018, that I have given up testing with Sanguino.

I have left the 644/644P ifdefs in the SdFat code.

Since SdFat is the base for the "native SD library" it will also break for the Sanguino if the Arduino group updates to the new version of SdFat.

@skyjumper: if you don't mind getting your hands a little dirty, you can update Sanguino "mostly" to 0022 pretty easily. I say "mostly" because the current WInterrupts.c no longer works with the '644P for hardware interrupts (use the one from 0020 or 0021 - probably not an issue if you don't use hardware interrupts; it will still work, but won't respond to INT1/INT2).

You can pretty much install 0022, then copy every file except pins_arduino.c and pins_arduino.h from the 0022 Arduino 'core' folder over the ones in your Sanguino core folder. Again, for hardware interrupt support, also grab WInterrupts.c from 0020 or 0021 and copy it over as well.

Thanks guys, I didn't realize I had selected a chip that's not really supported.

I'll try what you suggested. I have been using the Arduino 0022 environment with the 644P for a while now and this is the first time I had run into an issue. I'm not using hardware interupts myself, although I did have a plan to possibly use them. I can work around it though. If I do this right is there a chance that the SDFat library will work?

@Drmn4ea:

When you say "hardware interupt support" what exactly does that include? After looking through the source files in the core directory, I see HardwareSerial.cpp uses interupts to process incoming data into a ring buffer, which is a very standard way to service a serial port.

I ask because I am trying to understant what I might break.

Thanks...

Sorry, should have made that more clear. By "hardware interrupt" I meant the external interrupt lines (INT0/INT1/INT2). To be honest, I have not fully tested to what extent interrupt support works or does not work between 0022's WInterrupts.c and the '644PA - too many things on my plate right now! I know that interrupt-driven libraries like Serial and Wire were working, but when I tried putting the MCU to sleep and waking it from an INT pin, it will no longer wake/respond as of 0022. Other than this small issue (fixable by keeping the WInterrupts.c from 0020/0021), updating Sanguino all the way to 0022 has gone pretty smoothly for me!

Great news, thanks!

I gave it a try and was able to get SDFat to compile and run, and all seemed good, until... I tried to compile my own code. The compiler reported Serial1 as undefined. I went through the HardwareSerial.cpp/h files and found the proper macro, defined it, but no joy.

So I put the HardwareSerial.cpp/h files back from the original and that fixed Serial1 but then broke SDFat again. Sigh. Any ideas?

Thanks!

Okay I found some cores that work well with the 644P and the SDFat beta listed here compiles and the examples run well.

I'm wondering what the strategy for writing as fast as possible is? With Fat16 I would write to the file and keep track of how many packets I sent, and when I started to approach 512 bytes I would call sync(). I see this library also has a 512 byte buffer. Should I use the same strategy or will this library manage when to sync most efficiently?

also, I am wondering if it is necessary to call close()? This is a data logging app, which means the users are just going to cut power or yank the chip whenever they feel like it. Will that corrupt the file or will we just lose data in the write cache?

Thanks!

SdFat does not sync automatically. The cost of sync is very high and it is best to do it on a 512 byte boundary.

If you call sync in the middle of a block the current data block is written to the SD, the directory block is read, updated, and written. The data block must be reread during the next write. A sync costs 2048 bytes of IO in this case. You save 1024 bytes of I/O if you sync on a 512 byte boundary.

Sadly there is no simple option but to use sync when the user can just cut power. We don't do it on a PC, Linux, or Mac.

There is a way in SdFat to write at very high speed without sync but it is very complex. You can create a very large contiguous file and erase it with special calls. You then do raw writes to the file to avoid file system overhead. This is how I record audio in the WaveRP library Google Code Archive - Long-term storage for Google Code Project Hosting.. After the file is recorded I find the last recorded block of the file using a binary search and truncate the file to the correct size using the truncate() function.

This way I can recover the file if the user cuts power.

fat16lib:
SdFat does not sync automatically. The cost of sync is very high and it is best to do it on a 512 byte boundary.

That's what I thought, thank you for explaining.

So I'm confused by the code in the benchmark test:

 // do write test
  uint32_t n = FILE_SIZE/sizeof(buf);
  uint32_t t = millis();
  for (uint32_t i = 0; i < n; i++) {
    if (file.write(buf, sizeof(buf)) != sizeof(buf)) {
      error("write failed");
    }
  }
  file.sync();
  t = millis() - t;

You have a 100 byte buffer and you write it 10,000 times. You never call sync() until the end of the loop. If you didn't call sync() at all would you lose the entire data file or just the last 512 bytes?

It's worse than any of the above. If I didn't call sync, whole file is lost and the file system is corrupt since all the clusters in the file have been marked as allocated. Only sync() or close() update the directory entry with the file size.

If I had put

SdFile file;

inside setup like this

void setup() {
  SdFile file;

The file would be closed when setup ends, at least in the new beta version of SdFat, because the destructor is:

  ~SdFile() {close();}

This is why the "Arduino team" decided to set the O_SYNC flag in their SD.h wrapper for SdFat. This causes sync() to be called after every byte with print since print does single character writes. SD.h is a killer or at least an extreme test for the SD flash controller.

Many applications write a "record" of data and then call sync(). I just can't automatically do the "right thing" in the library.

fat16lib:
Many applications write a "record" of data and then call sync(). I just can't automatically do the "right thing" in the library.

Got it, thanks very much, that is a huge help. I'll do what I do with Fat16 then, track the amount of data written and then call sync() about every 512 bytes. If I understand correctly, this is the most efficient way to use the library, yes?

Last question, I hope. If i don't call close will I corrupt data? I know I'll lose all data since the last sync() call when the user hits the switch or pulls the card, but will the file and file system be okay other than loosing the last bit of data?

close() just calls sync() and marks the file closed so you won't lose data.

bool SdFile::close() {
  if (sync()) {
    type_ = FAT_FILE_TYPE_CLOSED;
    return true;
  }
  return false;
}

What works for Fat16 should work for SdFat.

Awesome! Thanks for your help, and for the library!

skyjumper:
I am having an issue. I can not get any of the sample apps to compile. The compiler complains of an error on line 40 of ArduinoStream.h: expected ')' before '&' token. I took a look and the reference parameter seemed to be declared correctly.

I tried several of the example files and each one gave me the same error. I'll try the SD classes without the stream IO and see if I have better luck.

I ran into the same problem today and found the solution to have nothing to do with Sanguino at all.
And why should it.

I am using a Diecemilia and got the same error.
The example code on the other hand compiled fine.

The solution was, not to use the Sketch->Import Library->(contributed)SdFat function of the IDE.
Wehn doing so, the IDE places #include statements for all headers in the libraries folder into the source of the sketch.
the #include <ArduinoStream.h> is at the top of the list in this case.

Simply removing all the #includes and adding
#include <SdFat.h>
#include <SdFile.h>
manually solved the problem. (for my sketch, anyway, I am using only those).

It puzzles me that you got the error when compiling the sample code, because that is only including the necessary headers, it seems.

And btw: Thanks to William for the excellent work.

hase

I designed SdFat so you only need

#include <SdFat.h>

for most sketches.

SdFatUtil.h is needed if you use FreeRam(), PgmPrint(), or PgmPrintln().

Including all the .h files can cost up to 3500 bytes of flash. The IDE scans these files and forces the loading of some functions that are not needed in the average sketch.

I will fix ArduinoStream.h so it doesn't produce a compile error but it should not be included at the start of a sketch.

Thanks for the feedback. I will add a caution to the documentation.