next topic »

[on:]

I am the author of SdFat, the base library for SD.  Several people have asked me why SD is so slow in Arduino 22 when you use print() for numbers.

Here is the reason SD is so slow and a way to speed it up by a factor of 100.

Print does character at a time writes when it formats numbers.  SD has been setup to do a flush after every write.  

This means that println(n) will call flush six times for a four digit number, four times for the digits and two times for the new line.

SD cards can only be read or written in 512 byte blocks and SdFat has a single 512 byte buffer.

A flush causes the data block to be written, the directory block to be read, updated and written.  The next write requires the data block to be read so it can be updated.  This is 2048 bytes of I/O.

Therefore writing the four byte number and a new line requires  12,288 bytes of I/O to the SD card.

This sketch writes 600 bytes to a SD card and takes 7.8 seconds for a rate of 77 bytes/sec.  The total I/O to the SD card is 1,228,800 bytes.

Code:

#include <SD.h>

File file;

void setup() {
  Serial.begin(9600);
  SD.begin(10);
  file = SD.open("WR_TEST1.TXT", FILE_WRITE);
 
  while(millis() < 1000);  // delay so mills() is four digits

  for (uint8_t i = 0; i < 100; i++) {
    file.println(millis());
  }
  file.close();
}

void loop() {}

If you replace FILE_WRITE with O_WRITE | O_CREAT and control when flush is called you can increase the speed by a large factor.

This sketch writes 600 bytes to the SD card in 0.070 seconds for a rate of 8,571 bytes/sec.

Code:

#include <SD.h>

File file;
void setup() {
  Serial.begin(9600);
  SD.begin(10);
  file = SD.open("WR_TEST2.TXT", O_CREAT | O_WRITE);

  while(millis() < 1000);  // delay so mills() is four digits
 
  for (uint8_t i = 0; i < 100; i++) {
    file.println(millis());
  }
  // not needed in this example since close() will call flush.
  file.flush();  
 
  file.close();
}
void loop() {}

[Reply #1 on:]

Thanks! More than interesting!

[Reply #2 on:]

Yes, thank you very much.

Is there a similar trick for the case in which it is desired to append to the end of the file?

-br
http://bitlash.net
http://entropymouse.com

[Reply #3 on:]

You can add O_APPEND to the open flags.  

If you need more features than the author of SD intended it may be better to use SdFat without the SD wrapper.

http://code.google.com/p/sdfatlib/

The SD wrapper library was written to simplify access to SdFat and has limited features.  I have neglected ease of use for SdFat.

I started work on Version 2 of SdFat about six months ago and one of the goals is ease of use for simple apps.

I decided to implement C++ iostreams functionality and restructure the internals of SdFat so it will still be a month or two before V2 is ready.

SdFat V2 will implement path names,  a current working directory, and a number of new classes in addition to the backward compatible SdFile class.

New classes include ifstream, ofstream, fstream and a class to provide cout functionality for Serial.

I have implemented an SdFormatter sketch that formats SD cards with a format that complies with the SD Association's specification and is optimal for use with the Arduino.

I have improved some of the high performance features.  I have been able to record 8-bit audio at 44.1 KB/sec using the Arduino's ADC.

[Reply #4 on:]

I started work on Version 2 of SdFat about six months ago and one of the goals is ease of use for simple apps.

This is great news. Hopefully will address a lot of the issues that appear on the forum related to SD card issues.

I decided to implement C++ iostreams functionality and restructure the internals of SdFat so it will still be a month or two before V2 is ready.

You're a tease. Keep us posted on your progress. I'm sure there are people willing to beta test software for you.

[Reply #5 on:]

I posted a demo of SdFat V2 here

http://code.google.com/p/beta-lib/downloads/list

A number of sketches that demo new features are included in the SdFat/examples folder.

Please post any comments or send them to the email address in the readme file.

[Reply #6 on:]

Hi - Ive been working with the arduino for a little while now and have tried to implement this technique into a project Im working on. When I have FILE_WRITE, all of my data gets recorded to the SD card as it should. When I swap that out for the O_WRITE | O_CREAT, then Im missing a lot of data that does not get saved. I tried adding a flush after every write command but that made no difference. Any ideas?

[Reply #7 on:]

If you are running the sketch several times you may want to use the O_APPEND flag.  Each time the sketch runs new data will be appended to the file.  This is how FILE_WRITE works, it appends to the file.

Here is an example.

Code:

#include <SD.h>
File file;
//------------------------
void setup() {
 Serial.begin(9600);
 
 if (!SD.begin()) {
   Serial.println("begin failed");
   return;
 }
 file = SD.open("TEST_SD.TXT", O_CREAT | O_APPEND | O_WRITE);
 
 file.println("Hello");

 // for test - better to close the file when finished
 file.flush();  
}
//--------------------------
void loop() {}

If you want to delete all data each time a sketch runs use the O_TRUNC flag with open to truncate the file to zero length:

Code:

file = SD.open("TEST_SD.TXT", O_CREAT | O_TRUNC |O_WRITE);

Here are all of the  open flag values for SdFat.  I am not sure they will all work with the Arduino SD wrapper.  You can download SdFat here http://code.google.com/p/sdfatlib/.

Open flag values are constructed by a bit-wise-inclusive OR of flags from the following list.

O_READ - Open for reading.

O_RDONLY - Same as O_READ.

O_WRITE - Open for writing.

O_WRONLY - Same as O_WRITE.

O_RDWR - Open for reading and writing.

O_APPEND - If set, the file offset shall be set to the end of the file prior to each write.

O_CREAT - If the file exists, this flag has no effect except as noted under O_EXCL below. Otherwise, the file shall be created

O_EXCL - If O_CREAT and O_EXCL are set, open() shall fail if the file exists.

O_SYNC - Call sync() after each write.  This flag should not be used with write(uint8_t), write_P(PGM_P), writeln_P(PGM_P), or the Arduino Print class. These functions do character at a time writes so sync() will be called after each byte.

O_TRUNC - If the file exists and is a regular file, and the file is successfully opened and is not read only, its length shall be truncated to 0.

[Reply #8 on:]

Hey that worked! Thank you very much!

EDIT - another small problem. My program went from about 0.45 seconds to function through down to about 0.23 with this alteration. That is acceptable to me, however, after letting the program run, I noticed that after about 1500 seconds or so, the program will hit a few loops where it tends to slow down, back to almost 0.5 seconds to function each time. Then it will resume its normal pace. This happens on and off every ~10 seconds. While the program slows down, I have noticed the L light on the arduino (hooked to pin 13 I believe) is staying on for longer than usual. I am led to believe that there is a problem that the writing to the sd card is slowing it down, but Im not sure why it would work fine for the rest of the time and then slow down for a few instances.

[Reply #9 on:]

One question about this SD library... is it possible to name the files created on the SD card by a variable...
something like...


File file;
int date;

void loop()
{
if (!(date==getdate())) //something that is excluded everyday
{
date=getdate();  //date collected from a Real time clock module
}
...some code...

file = SD.open(date, O_CREAT | O_WRITE | O_APPEND); //data written to the a file on the SD card named with the current date

rest of the code....
}

[Reply #10 on:]

Is it possible to create a file on the SD card with a variable name. (a name which is desided be the program)

so instead of
file = SD.open("Test.txt", O_CREAT | O_WRITE);

I would like something like...

int variable;
file = SD.open(variable, O_CREAT | O_WRITE);
.. and the variable changes from time to time so there will be created many files on the SD card, I was thinkin of using this for logging data from a meter, and have new file for each day / each week / each hour...

[Reply #11 on:]

Is it possible to create a file on the SD card with a variable name.

Of course it is, provided that the variable is of the correct type (char array) and size ("ThisHereBeMyNewLongAndCompilicatedFileName.WithReallyLongExtension" isn't).

[Reply #12 on:]

ohhh I was just figureing out what I did wrong

here is a part of my test code

char myString[glow]s[/glow][]="Test.txt";
void loop()
{
  file = SD.open([glow]myString[/glow], O_CREAT | O_WRITE | O_APPEND); //forgot on "s" myString[glow]s[/glow]

  for (int i = 0; i < 5000; i++) //writes numbers incrementing from 0-4999 to the SD card
    {
    file.println(i);
    }
  file.flush();
  file.close();
}

now it works fine