Go Down

Topic: Urgent: SD card write speeds (Read 5154 times) previous topic - next topic

fore4runner

Hi there,

I'm working on a datalogger that basically follows the exact protocol laid out in the Datalogger example.  However, this program maxes out at a sampling rate of 5 Hz due to the long time taken by the dataFile.println(dataString); line.

However, the SPI communication should be happening at nearly 4 MHz, why is the program taking nearly 200 ms to transfer only 96 bits of information when I suspect that at 4 MHz clock rate the data should only take 25 nS  (maybe up to 200 nS depending on the handshaking required).

If anyone has any tips on how to reduce the write times that would be a great help to me as I am looking to get my sampling rates up to at least 500 Hz.

Thanks so much for your time,

Andre Bezanson

Aeturnalus

Send your samples in larger blocks: Flash requires that you write a few kb at once (4kb blocks, iirc), so you should just wait until you've got enough data to fill at least half a block.  It'll be much faster.

robtillaart

Aeturnalus is right, writing in large blocks is often more efficient.

Do you have a link of the datasheet/spec of the datalogger?

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

PaulS

Quote
If anyone has any tips on how to reduce the write times that would be a great help to me as I am looking to get my sampling rates up to at least 500 Hz.

Just the usual one that I make whenever anyone asks how to improve code that they haven't posted. Post your code.

fore4runner

#4
Jun 05, 2011, 05:02 pm Last Edit: Jun 05, 2011, 05:08 pm by fore4runner Reason: 1
Hi there,

Sorry for not posting the code or stating more about the project.  I'm using a ADXL335 as the accelerometer with the corner frequency set to 250 Hz (therefore I need to sample with at least 500Hz to get prevent aliasing).  There is a typical hookup to a micro SD card and the device is packaged such that it fits nicely onto a arduino pro.  I power the unit from 3.7 volt lithium cells, which are then passed through a step-up converter which I am using to power the 5v version of the arduino pro (I used the 5v version because the higher clock speed allows for faster communication).  A 3.3 volt regulator is then used to provide power for the adxl335 and the SD card.

Basically this board is a refined version of an earlier unit were I was using the sparkfun breakout boards for the ADXL335, the regulator and an openlog sd writer.  With the earlier unit I was able to get write speeds up to 500  Hz with the openlog so I think I should be able to get this just programming the card directly (possibly even much faster).

Here is a picture of the device (sorry about the poor quality of the image):

http://imageshack.us/photo/my-images/339/cimg0001jy.jpg/


The code is the exact datalogger example with a couple additions (see below).  By toggling a pin high and low I've found that the line: logfile.println(dataString); takes about 200ms which is killing my write times.  I'd like to be able to have continuous monitoring (ie I'd like to avoid reading for a couple seconds then writing for a second as it's fairly important for the application that the sample period is constant).

I'll try playing around with the write sizes now and I'll let you guys know how that goes, thanks again for the tips!


Code: [Select]
/*
 SD card file dump

This example shows how to read a file from the SD card using the
SD library and send it over the serial port.

The circuit:
* SD card attached to SPI bus as follows:
** MOSI - pin 11
** MISO - pin 12
** CLK - pin 13
** CS - pin 10

created  22 December 2010

This example code is in the public domain.

*/

#include <SD.h>

#define redLEDpin 3
#define greenLEDpin 4
#define SPIPORT PORTB
#define SPIDDR  DDRB
#define MOSI 11       // Arduino pin 11
#define MISO 12       // Arduino pin 12
#define SCLK 13       // Arduino pin 13
#define CS 10         // Arduino pin 9
#define SS 2         // Arduino pin 10

#define SB(x,n) (x|=(1<<n))  // set bit
#define CB(x,n) (x&=~(1<<n)) // clear bit


const int xpin = A2;                  // x-axis of the accelerometer
const int ypin = A1;                  // y-axis
const int zpin = A0;                  // z-axis (only on 3-axis models)
const int flasher = 9;             // analog input pin 4 -- ground

int sensor = 0;
byte clr;


// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.
const int chipSelect = CS;
//char filename[] = "FISHRUN00.TXT";
 char filename[] = "LOGGER00.TXT";

File logfile;

void error(char *str)
{
 Serial.print("error: ");
 Serial.println(str);
 
 // red LED indicates error
 digitalWrite(redLEDpin, HIGH);
 
 while(1);
}

void spi_init(void)
{
 // As Master, all SPI pins go out except MOSI which comes in
 SB(SPIDDR,MOSI);
 SB(SPIDDR,SCLK);
 SB(SPIDDR,CS);
 SB(SPIDDR,SS);
 CB(SPIDDR,MISO);

 SB(SPIPORT,SS);
 CB(SPIPORT,CS);  // Activate card

 // spi enabled, master mode, clock @ f/128 for init
 SPCR = (1<<SPE)|(1<<MSTR);//|(0<<SPR1)|(0<<SPR0);

 // dummy read registers to clear previous results
 clr=SPSR;
 clr=SPDR;

 // you might choose to end this function with a delay
 // to ensure that the SPI hardware is ready before
 // attempting any SPI transmissions
}



void setup()
{
//  SPCR
//| 7    | 6    | 5    | 4    | 3    | 2    | 1    | 0    |
//| SPIE | SPE  | DORD | MSTR | CPOL | CPHA | SPR1 | SPR0 |
//  SPCR = (1<<SPE)|(1<<MSTR);
//  clr=SPSR;
//  clr=SPDR;
 spi_init();
 Serial.begin(9600);
 Serial.print("Initializing SD card...");
 // make sure that the default chip select pin is set to
 // output, even if you don't use it:
 pinMode(10, OUTPUT);
 pinMode(flasher, OUTPUT);
 digitalWrite(flasher, LOW);
 
 // see if the card is present and can be initialized:
 if (!SD.begin(chipSelect)) {
   Serial.println("Card failed, or not present");
   // don't do anything more:
   return;
 }
 Serial.println("card initialized.");
 
 // open the file. note that only one file can be open at a time,
 // so you have to close this one before opening another.
 // create a new file
 
 
 // create a new file

 for (uint8_t i = 0; i < 100; i++) {
   filename[6] = i/10 + '0';
   filename[7] = i%10 + '0';
   if (! SD.exists(filename)) {
     // only open a new file if it doesn't exist
     logfile = SD.open(filename, FILE_WRITE);
     break;  // leave the loop!
   }
 }
 
 if (! logfile) {
   error("couldnt create file");
 }
 
 Serial.print("Logging to: ");
 Serial.println(filename);
 
 


}

void loop()
{

   logfile = SD.open(filename, FILE_WRITE);
   
    String dataString = "";
   
   // read three sensors and append to the string:
     
     sensor = analogRead(xpin);
     dataString += String(sensor);
     
     dataString += "\t";
     
     sensor = analogRead(ypin);
     dataString += String(sensor);
     
     dataString += "\t";
     
     sensor = analogRead(zpin);
     dataString += String(sensor);
     
     digitalWrite(flasher, HIGH);
     logfile.println(dataString);
     logfile.close();
 //    delay(100);
     digitalWrite(flasher, LOW);


 //    delay(100);
     
 //    Serial.println(dataString);
   
}



robtillaart


instead of strings you could use an array of chars, more memory efficient.

Code: [Select]
   
void loop()
{
       String dataString = "";
      // read three sensors and append to the string:
      sensor = analogRead(xpin);
      dataString += String(sensor);
      dataString += "\t";
      sensor = analogRead(ypin);
      dataString += String(sensor);
      dataString += "\t";
      sensor = analogRead(zpin);
      dataString += String(sensor);

      digitalWrite(flasher, HIGH);
      logfile.println(dataString);
      logfile.close();
      digitalWrite(flasher, LOW);
}


could be:

Code: [Select]

void loop()
{
  #define SIZE 100            // experiment with the size
  int samples[SIZE ][3];

  // MAKE MEASUREMENTS
  int i=0;
  for (i=0; i<SIZE ;i++)
  {
    samples[1][0] = analogRead(xpin);
    samples[1][1] = analogRead(ypin);
    samples[1][2] = analogRead(zpin);
    // optional delay
  }


  // LOG TO FILE
  unsigned long t = millis();

  digitalWrite(flasher, HIGH);
  logfile = SD.open(filename, FILE_WRITE);   //  FILE_APPEND  ?????

  for (i=0; i<SIZE ;i++)
  {
    // CREATE OUTPUT STRING
    char buffer[20];   // increase if needed
    sprintf(buffer,"%d\t%d\t%d", samples[i][0],samples[i][1],samples[i][2], );   // max 15 chars
    logfile.println(buffer);
  }
  logfile.close();
  digitalWrite(flasher, LOW);
  // optional delay

  Serial.print("writing data (millis):  ");
  Serial.println(millis() - last);

}


To optimize writing make an array of 100 x 3 ints
int samples[100][3];
and fill it up with data. if the array is full, write all 100 measurements to the SDcard
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

PaulS

Is opening and closing the file on every pass through loop() necessary? How long does that take?

Because you do not keep track of where in the file the last write occurred, the File function must find the end of the file, so it can append to the file, each time you write to the file.

Depending on how critical it is to not lose data, you might close and open the file every 5th write or every 10th write, or every 100th write.

fore4runner


Is opening and closing the file on every pass through loop() necessary? How long does that take?

Because you do not keep track of where in the file the last write occurred, the File function must find the end of the file, so it can append to the file, each time you write to the file.

Depending on how critical it is to not lose data, you might close and open the file every 5th write or every 10th write, or every 100th write.


Believe it or not, it makes almost no difference.  It probably adds a couple milliseconds but as the data takes so long to write it is only a marginal issue (though that could be an issue once I get past the current issue).

Cheers

Aeturnalus

Sample in discrete chunks of time: like I said before, SD cards (and other similar flash devices) must be written to in blocks if you want a faster IO speed.  Also, don't open and close the file every time you go through the loop: open it in setup(), and close it on some external input (the file will corrupt if you don't close it before removing power and/or removing the SD card, but you can, say, press a button to close the file and then remove the card). 

Normally, open/close operations are less efficient than writing, although I've never used the SD library myself.

fore4runner

Unfortunately, I cannot do the discrete chunks of time technique as it leads to an inconsistent sampling rate (not acceptable for this application)  also the device is designed to sample until the battery dies so I'm afraid that the opening and closing might be necessary (though from my tests with the oscilloscope this isn't a problem as it doesn't take that long for the opening and closing routines).

The point that I don't understand is that I could get the openlog device to write up to 500 Hz.  Therefore the openlog device had to be able to do things more efficiently then is possible using the standard library.  If anyone knows how they may have accomplished this I would appreciate an explanation as my programming skills are not to the point where I can reverse engineer their code.

fore4runner

That sounds like an awesome idea KE7GKP!  Any idea what functions I'd use for that? 

PS I assume a block is 512 bytes?

Cheers

fore4runner

That does sound like an ideal solution, unfortunately my programming skills aren't yet at the level of being able to write a library file. 

For analysis of the data a text file is the most convenient so I'll probably just reduce the 10 bit ADC values to base 32 values as a way of limiting the number of ascii characters. 

Thanks for the advice!  I'm still checking out all the different library's that I can find hoping I find one with a faster write time for small data chunks though it seems like you guys are correct about that not being an option.  Though the openlog seems to do it somehow..........

fat16lib

#12
Jun 05, 2011, 11:35 pm Last Edit: Jun 05, 2011, 11:36 pm by fat16lib Reason: 1
It is possible to log data to a file on an SD card at higher rates.  I wrote a library to record sound at 44100 8-bit samples per seconds using the Arduino's ADC.

This library is here http://code.google.com/p/waverp/.

To do this I do raw writes to a contiguous file using two 512 byte buffers.  I read the ADC and buffer the data in an interrupt routine driven by timer 1.

The library is somewhat complex since it records and plays .wav files.  It would be much simpler to just log data.

There is an example in my SdFat library that illustrates the use of raw write functions in SdFat.  This example is in the examples/SdFatRawWrite folder.

SdFat is here http://code.google.com/p/sdfatlib/.

It is important to use an SD card that has short write latency in SPI mode.  I have good luck with cheap blue SanDisk 1G or 2G cards formatted FAT16 with 32 KB clusters.  Don't buy a pricy class 10 SDHC card, they often don't preform well in SPI mode.

You can use the above example, SdFatRawWrite, to test your card.



Aeturnalus

The openlog writes in large blocks (RXBUFFER/2) =).  Since it's open source, you can easily just take a look at its source and use the same strategies.  In this case, it waits until it's reached a certain amount of data in its write buffer, and then syncs it to the card (what we've been telling you to do for all this time...). 

Coincidentally, it uses the SdFatlib mentioned by fat16lib. 

fore4runner

Thanks for that Aeturnalus!  I just realized that with my previous version there were holes in the data where the openlog device, stopped listening to the serial information (being recorded at the 500Hz sample rate) and it switched over to the write mode. 

So yes, I completely agree now that sending the data in large chunks is the only way I can get anywhere's close to the write speeds I'm looking for.

I've been working though the examples in the SdFatlib resource and it's really good!  I've got a very limited understanding of programming techniques so it is a huge help!

The sampling rate of your program is much higher then what I need as I only need 3(sensors)*500(samples per second)*10(bits) of data or 15000 bits versus your 352800!  So basically I'm going to switch over from what I was working on and look really hard at trying to convert your program to what I need.  Thanks again for posting that.

Go Up