Urgent: SD card write speeds

Hi there,

I'm working on a datalogger that basically follows the exact protocol laid out in the Datalogger example. However, this program maxes out at a sampling rate of 5 Hz due to the long time taken by the dataFile.println(dataString); line.

However, the SPI communication should be happening at nearly 4 MHz, why is the program taking nearly 200 ms to transfer only 96 bits of information when I suspect that at 4 MHz clock rate the data should only take 25 nS (maybe up to 200 nS depending on the handshaking required).

If anyone has any tips on how to reduce the write times that would be a great help to me as I am looking to get my sampling rates up to at least 500 Hz.

Thanks so much for your time,

Andre Bezanson

Send your samples in larger blocks: Flash requires that you write a few kb at once (4kb blocks, iirc), so you should just wait until you've got enough data to fill at least half a block. It'll be much faster.

Aeturnalus is right, writing in large blocks is often more efficient.

Do you have a link of the datasheet/spec of the datalogger?

If anyone has any tips on how to reduce the write times that would be a great help to me as I am looking to get my sampling rates up to at least 500 Hz.

Just the usual one that I make whenever anyone asks how to improve code that they haven't posted. Post your code.

Hi there,

Sorry for not posting the code or stating more about the project. I'm using a ADXL335 as the accelerometer with the corner frequency set to 250 Hz (therefore I need to sample with at least 500Hz to get prevent aliasing). There is a typical hookup to a micro SD card and the device is packaged such that it fits nicely onto a arduino pro. I power the unit from 3.7 volt lithium cells, which are then passed through a step-up converter which I am using to power the 5v version of the arduino pro (I used the 5v version because the higher clock speed allows for faster communication). A 3.3 volt regulator is then used to provide power for the adxl335 and the SD card.

Basically this board is a refined version of an earlier unit were I was using the sparkfun breakout boards for the ADXL335, the regulator and an openlog sd writer. With the earlier unit I was able to get write speeds up to 500 Hz with the openlog so I think I should be able to get this just programming the card directly (possibly even much faster).

Here is a picture of the device (sorry about the poor quality of the image):

The code is the exact datalogger example with a couple additions (see below). By toggling a pin high and low I've found that the line: logfile.println(dataString); takes about 200ms which is killing my write times. I'd like to be able to have continuous monitoring (ie I'd like to avoid reading for a couple seconds then writing for a second as it's fairly important for the application that the sample period is constant).

I'll try playing around with the write sizes now and I'll let you guys know how that goes, thanks again for the tips!

/*
  SD card file dump
 
 This example shows how to read a file from the SD card using the
 SD library and send it over the serial port.
 	
 The circuit:
 * SD card attached to SPI bus as follows:
 ** MOSI - pin 11
 ** MISO - pin 12
 ** CLK - pin 13
 ** CS - pin 10
 
 created  22 December 2010
 
 This example code is in the public domain.
 	 
 */

#include <SD.h>

#define redLEDpin 3
#define greenLEDpin 4
#define SPIPORT PORTB
#define SPIDDR  DDRB
#define MOSI 11       // Arduino pin 11
#define MISO 12       // Arduino pin 12
#define SCLK 13       // Arduino pin 13
#define CS 10         // Arduino pin 9
#define SS 2         // Arduino pin 10

#define SB(x,n) (x|=(1<<n))  // set bit
#define CB(x,n) (x&=~(1<<n)) // clear bit


const int xpin = A2;                  // x-axis of the accelerometer
const int ypin = A1;                  // y-axis
const int zpin = A0;                  // z-axis (only on 3-axis models)
const int flasher = 9;             // analog input pin 4 -- ground

int sensor = 0;
byte clr;


// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.
const int chipSelect = CS;
//char filename[] = "FISHRUN00.TXT";
  char filename[] = "LOGGER00.TXT";

File logfile;

void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  
  // red LED indicates error
  digitalWrite(redLEDpin, HIGH);
  
  while(1);
}

void spi_init(void)
{
  // As Master, all SPI pins go out except MOSI which comes in
  SB(SPIDDR,MOSI);
  SB(SPIDDR,SCLK);
  SB(SPIDDR,CS);
  SB(SPIDDR,SS); 
  CB(SPIDDR,MISO);

  SB(SPIPORT,SS); 
  CB(SPIPORT,CS);  // Activate card

  // spi enabled, master mode, clock @ f/128 for init
  SPCR = (1<<SPE)|(1<<MSTR);//|(0<<SPR1)|(0<<SPR0);

  // dummy read registers to clear previous results
  clr=SPSR;
  clr=SPDR;

  // you might choose to end this function with a delay
  // to ensure that the SPI hardware is ready before
  // attempting any SPI transmissions
}



void setup()
{
//  SPCR
//| 7    | 6    | 5    | 4    | 3    | 2    | 1    | 0    |
//| SPIE | SPE  | DORD | MSTR | CPOL | CPHA | SPR1 | SPR0 |
//  SPCR = (1<<SPE)|(1<<MSTR);
//  clr=SPSR;
//  clr=SPDR;
  spi_init();
  Serial.begin(9600);
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  pinMode(flasher, OUTPUT);
  digitalWrite(flasher, LOW); 
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");
  
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  // create a new file
  
  
  // create a new file

  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = i/10 + '0';
    filename[7] = i%10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE); 
      break;  // leave the loop!
    }
  }
  
  if (! logfile) {
    error("couldnt create file");
  }
  
  Serial.print("Logging to: ");
  Serial.println(filename);
  
  


}

void loop()
{

    logfile = SD.open(filename, FILE_WRITE);
    
     String dataString = "";
     
    // read three sensors and append to the string:
      
      sensor = analogRead(xpin);
      dataString += String(sensor);
      
      dataString += "\t"; 
      
      sensor = analogRead(ypin);
      dataString += String(sensor);
      
      dataString += "\t"; 
      
      sensor = analogRead(zpin);
      dataString += String(sensor);
      
      digitalWrite(flasher, HIGH);
      logfile.println(dataString);
      logfile.close();
  //    delay(100);
      digitalWrite(flasher, LOW);


  //    delay(100);
      
  //    Serial.println(dataString);
    
}

instead of strings you could use an array of chars, more memory efficient.

void loop()
{
       String dataString = "";
      // read three sensors and append to the string:
      sensor = analogRead(xpin);
      dataString += String(sensor);
      dataString += "\t"; 
      sensor = analogRead(ypin);
      dataString += String(sensor);
      dataString += "\t"; 
      sensor = analogRead(zpin);
      dataString += String(sensor);

      digitalWrite(flasher, HIGH);
      logfile.println(dataString);
      logfile.close();
      digitalWrite(flasher, LOW);
}

could be:

void loop()
{
  #define SIZE 100            // experiment with the size
  int samples[SIZE ][3];

  // MAKE MEASUREMENTS
  int i=0;
  for (i=0; i<SIZE ;i++)
  {
    samples[1][0] = analogRead(xpin);
    samples[1][1] = analogRead(ypin);
    samples[1][2] = analogRead(zpin);
    // optional delay
  }


  // LOG TO FILE
  unsigned long t = millis();

  digitalWrite(flasher, HIGH);
  logfile = SD.open(filename, FILE_WRITE);   //  FILE_APPEND  ?????

  for (i=0; i<SIZE ;i++)
  {
    // CREATE OUTPUT STRING
    char buffer[20];   // increase if needed 
    sprintf(buffer,"%d\t%d\t%d", samples[i][0],samples[i][1],samples[i][2], );   // max 15 chars
    logfile.println(buffer);
  }
  logfile.close();
  digitalWrite(flasher, LOW);
  // optional delay

  Serial.print("writing data (millis):  ");
  Serial.println(millis() - last);

}

To optimize writing make an array of 100 x 3 ints
int samples[100][3];
and fill it up with data. if the array is full, write all 100 measurements to the SDcard

Is opening and closing the file on every pass through loop() necessary? How long does that take?

Because you do not keep track of where in the file the last write occurred, the File function must find the end of the file, so it can append to the file, each time you write to the file.

Depending on how critical it is to not lose data, you might close and open the file every 5th write or every 10th write, or every 100th write.

PaulS:
Is opening and closing the file on every pass through loop() necessary? How long does that take?

Because you do not keep track of where in the file the last write occurred, the File function must find the end of the file, so it can append to the file, each time you write to the file.

Depending on how critical it is to not lose data, you might close and open the file every 5th write or every 10th write, or every 100th write.

Believe it or not, it makes almost no difference. It probably adds a couple milliseconds but as the data takes so long to write it is only a marginal issue (though that could be an issue once I get past the current issue).

Cheers

Sample in discrete chunks of time: like I said before, SD cards (and other similar flash devices) must be written to in blocks if you want a faster IO speed. Also, don't open and close the file every time you go through the loop: open it in setup(), and close it on some external input (the file will corrupt if you don't close it before removing power and/or removing the SD card, but you can, say, press a button to close the file and then remove the card).

Normally, open/close operations are less efficient than writing, although I've never used the SD library myself.

Unfortunately, I cannot do the discrete chunks of time technique as it leads to an inconsistent sampling rate (not acceptable for this application) also the device is designed to sample until the battery dies so I'm afraid that the opening and closing might be necessary (though from my tests with the oscilloscope this isn't a problem as it doesn't take that long for the opening and closing routines).

The point that I don't understand is that I could get the openlog device to write up to 500 Hz. Therefore the openlog device had to be able to do things more efficiently then is possible using the standard library. If anyone knows how they may have accomplished this I would appreciate an explanation as my programming skills are not to the point where I can reverse engineer their code.

That sounds like an awesome idea KE7GKP! Any idea what functions I'd use for that?

PS I assume a block is 512 bytes?

Cheers

That does sound like an ideal solution, unfortunately my programming skills aren't yet at the level of being able to write a library file.

For analysis of the data a text file is the most convenient so I'll probably just reduce the 10 bit ADC values to base 32 values as a way of limiting the number of ascii characters.

Thanks for the advice! I'm still checking out all the different library's that I can find hoping I find one with a faster write time for small data chunks though it seems like you guys are correct about that not being an option. Though the openlog seems to do it somehow..........

It is possible to log data to a file on an SD card at higher rates. I wrote a library to record sound at 44100 8-bit samples per seconds using the Arduino's ADC.

This library is here Google Code Archive - Long-term storage for Google Code Project Hosting..

To do this I do raw writes to a contiguous file using two 512 byte buffers. I read the ADC and buffer the data in an interrupt routine driven by timer 1.

The library is somewhat complex since it records and plays .wav files. It would be much simpler to just log data.

There is an example in my SdFat library that illustrates the use of raw write functions in SdFat. This example is in the examples/SdFatRawWrite folder.

SdFat is here http://code.google.com/p/sdfatlib/.

It is important to use an SD card that has short write latency in SPI mode. I have good luck with cheap blue SanDisk 1G or 2G cards formatted FAT16 with 32 KB clusters. Don't buy a pricy class 10 SDHC card, they often don't preform well in SPI mode.

You can use the above example, SdFatRawWrite, to test your card.

The openlog writes in large blocks (RXBUFFER/2) =). Since it's open source, you can easily just take a look at its source and use the same strategies. In this case, it waits until it's reached a certain amount of data in its write buffer, and then syncs it to the card (what we've been telling you to do for all this time...).

Coincidentally, it uses the SdFatlib mentioned by fat16lib.

Thanks for that Aeturnalus! I just realized that with my previous version there were holes in the data where the openlog device, stopped listening to the serial information (being recorded at the 500Hz sample rate) and it switched over to the write mode.

So yes, I completely agree now that sending the data in large chunks is the only way I can get anywhere's close to the write speeds I'm looking for.

I've been working though the examples in the SdFatlib resource and it's really good! I've got a very limited understanding of programming techniques so it is a huge help!

The sampling rate of your program is much higher then what I need as I only need 3(sensors)*500(samples per second)*10(bits) of data or 15000 bits versus your 352800! So basically I'm going to switch over from what I was working on and look really hard at trying to convert your program to what I need. Thanks again for posting that.

As a followup to this thread. I was able to get the write speeds I needed by modifying the 'bench' example provided with the sdfat library to write what I wanted. It basically did what everyone was suggesting with writing larger chunks of data, though it was nice having an example as my programming skills are weak.

In the future I hope to get into the example code that Aeturnalus provided as it looks to be much much more efficient then what I am doing now.

Cheers, and thanks for all the help everyone!

Hi Fore4Runner,

i would love to see your adapted logging code as I'm trying to achieve a similar optimization!

I want to log the events on a CAN Bus, which means that I need to be able to log up to 4 events per milisecond ...

At the moment, the best I achieve is doing 10 consecutive writes with 60 microsecond for each write operation and then having one operation taking between 5000-7000 microseconds. Although i haven't optimized it that way, I suppose, that that's when the actual writing to the SD card occurs.
(btw, the actual code before the writing adds 400 microseconds due tu a sprintf operation ... but i haven't had a better idea yet to put together a string with 10 different values... )

any help is greatly appreciated,
Keija

@Keija,
you can post your code too if you want the forum to have a look....

sure, that would be great!

I'm currently using the following SD library:
http://arduino.cc/forum/index.php/topic,59287.0.html
as i hoped, that it would improve write speeds... It did, but only by a margin

this is where the magic happens

  // create the String to be written
  sprintf(strtest,"%lu-%u:%hx,%hx,%hx,%hx,%hx,%hx,%hx,%hx;\r\n",t,testId,data[0],data[1],data[2],data[3],data[4],data[5],data[6],data[7]);
  
// write the string
  file.write(strtest);

and the "full" SD part

  void setup() {

	...

  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!card.init(SPI_FULL_SPEED,9)) error("card.init failed");

  // initialize a FAT volume
  if (!volume.init(&card)) error("volume.init failed");

  // open the root directory
  if (!root.openRoot(&volume)) error("openRoot failed");

  
  // create a new file
  char name[] = "RAWLOG00.TXT";
  for (uint8_t i = 0; i < 100; i++) {
  	name[6] = i/10 + '0';
  	name[7] = i%10 + '0';
  	if (file.open(&root, name, O_CREAT | O_EXCL  | O_WRITE)) break;
  }
  if (!file.isOpen()) error ("file.create"); 

...

}



uint8_t data[8];
int testId = 123;


void loop(){

// this is just to show the type of data, that is used
  data[0] = B00000001;
  data[1] = B00000011;
  data[2] = B00000101;
  data[3] = B00001001;
  data[4] = B00010001;
  data[5] = B00100001;
  data[6] = B01000001;
  data[7] = B10000001;
  
  
  char strtest [52];
 
  uint32_t tw = micros();
       
  // create the String to be written
  sprintf(strtest,"%lu-%u:%hx,%hx,%hx,%hx,%hx,%hx,%hx,%hx;\r\n",t,testId,data[0],data[1],data[2],data[3],data[4],data[5],data[6],data[7]);
  
// write the string
  file.write(strtest);


  uint32_t tw2 = micros();
  
  uint32_t diff = tw2- tw;

 // print the time it took to write the string and to sd
 Serial.println(diff );

}

the uInt data array is only for demo purposes, but it shows the type of data, that i want to write ...

at the moment, im writing the data in the following format
time-id:data1,data2 ....,data7;
time2-id ....

this data is then parsed in a java program - writing raw data could work, as i don't necessarily need the log to be humanly readable.

I've already spent 2 days trying to optimize my code - so any hint would be greatly appreciated!

Just wanted to report, that i've achieved a !massive! improvement (x10) and that I will share my results with you tomorow!!

thanks to anyone, who's taken a look at my problem