Need read/write times for SD lib to be faster

You will never achieve reliable data logging at 1 ksps with the SD.h wrapper. It is possible to log data at high rates with SdFat but it requires special techniques. The problem is latency and buffering, not the SPI bus.

I wrote the WaveRP library to record audio at up to 44.1 ksps for 8-bit data. It is based on SdFat but is complex. See the new beta here Google Code Archive - Long-term storage for Google Code Project Hosting.

It is possible to log data at high rates with SdFat but it requires special techniques. The problem is latency and buffering, not the SPI bus.

What special techniques would I need to implement to get a higher read/write times? Do I need to change the buffer size?

Here is one trick I read from another post here:

Use multiple EEPROMS. EEPROMs require much less CPU for access so it's faster. When you gang them up, then write to ROM1, then to ROM2, then ROM3... you save time since you don't have to wait till ROM1 finishes writing. You have four of them together (very easy to wire, no surface mount needed), do your measurements, write to ROM1, do measurements again, write to ROM2, so on. It may help speed up your process quite a bit when all you have to do is transferring data and the ROM does the writing. Using sd cards is quite cpu intensive.

Sd.h should be able to handle 1ks/s quite easily. In fact, I'm currently doing 2ks/s + anothre 500/s via xbee, and dumping it all to an SD card in real time.

All you really need to do is create a buffer. Writing 2 bytes at a time is EXTREMELY slow.

void * temp =malloc(BUFFER_SIZE * sizeof (unsigned short));
if(temp != null)
  sampleBuffer = (unsigned short*) temp;
else{ 
  //Out of memory
}

fill your buffer with samples, then write the entire thing at once. Writing a buffer of 100 Bytes doesn't take much longer then writing a single value.

You can write at an average rate of well over 2 KB per second with a buffer but this is not 1 ksps with two byte samples. You will miss samples unless you capture data in an IRS or using a device with buffering like the xbee.

The reason is that SdFat which SD.h is bases on will take far more than a millisecond to do a write when a new cluster is allocated to the file.

SD cards are designed for sequential writes so accessing the two copies of the file allocation table causes long latency. 20 - 30 ms is typical for allocating a cluster.

To record audio with no missed samples I use two buffers and and capture data in a timer driven IRS.

Does the SD card class make a difference or is it the hardware most Arduino users use? Could we get full speed from a class 10 card?

I'm still a little unsure on how to implement the buffers. Currently I have an output compare interrupt on the 16 bit timer that allows me to sample at 1kHz and I am just writing the ADC data to the SD card every time the ISR fires. To implement the buffer system you described, do I just write to one buffer and when it fills write it to the SD card while the sampling continues to fill up a second buffer then just alternate like that? I'm also unsure how to write an entire buffer at once.

The card class is not important. You need very large multiple block writes to get the advantage of class 10 cards.

Yes, fill one buffer in the interrupt routine while the other is being written. For best results ues buffers with a size of 128, 256, or 512 bytes. This works best since a buffer will align on SD block boundries.

I think I'm thinking about writing the buffers to the SD card incorrectly... It can't be that I write each value incrementally, that would be the same thing as writing each as it came in from the ADC, right? I've been able to get write times to ~190 microseconds (~10kB/sec) using:

char buff[4];
sprintf(buff, "%i",<value from adc>);
myFile.write(buff);
myFile.write("\r\n");

This is significantly faster than the println route. Can you help me on how to write an entire buffer at once, without blocking the ISR. It is critical that I do not lose samples.

Maybe the eaisest way to avoid missed samples is to use a ring buffer like Serial. Put the binary ADC values in the buffer in the ISR and remove and format the values in the background.

Using sprintf is faster than Arduino print. I suggested an improved smaller version of print that is faster and smaller but I guess the Arduino group has no intrest.

You should use the format string "%i\r\n" so you can use one write.

Look at the Arduino hardware serial driver for details on using the ring buffer.

Hey, I was just testing using the "%i\r\n in sprintf and I noticed that it actually takes shorter to write when that is done than when you break it up into 2 writes like I had originally posted it. By shorter I mean 5 microseconds, but the bigger problem I ran into with sprintf(buff,"%i\r\n",analogRead(0)) was that it often inserted characters in between readings instead of new lines. I'm also reading the values off to an android phone through bluetooth and this really threw a wrench in the software I had quickly written up to read the values out. Do you have any idea why it would be doing this? I tried echoing out the data being written to the SD through the UART and found the same errors so it wasn't an error in the writing to the SD.

but the bigger problem I ran into with sprintf(buff,"%i\r\n",analogRead(0)) was that it often inserted characters in between readings instead of new lines.

Sounds to me like buff isn't big enough. The %i format statement will require up to 4 characters to hold the 1023 maximum value returned by analogRead. The \n, \r, and trailing NULL add three more, so buff needs to be at least 7 characters long. How big is it?

You're exactly right. I had the char buffer at 4 when I was just writing the 4 characters from the ADC. I forgot to add some more for the escape characters.

Hi,
I have 3 sensors attached to a n Arduino board with atmega 328 that I am using for a motion capture project. The sensors are and ADX345 accelerometer, HMC 5883L magnetometer and IDG3200 gyro. I have the sensors working correctly and I can sample them just fine. The problem I have is writing to the micro SD. I'd like to be able to write the data at a rate of about 200Hz. Does anyone have he code to do this?

The problem I have is writing to the micro SD. I'd like to be able to write the data at a rate of about 200Hz.

Two bytes at 200Hz or 2Kb at 200Hz. Frequency alone is a useless value.

mokwichi,

You will need special buffering to log at 200 samples/sec. I have posted a number of examples here Google Code Archive - Long-term storage for Google Code Project Hosting..

The examples in fastLoggerBeta20110802.zip use custom timer ISRs and buffering.

The examples in ChibiOS20111027.zip and FreeRTOS20111031.zip use a real-time kernel to run two threads. One thread reads sensors at fixed time intervals and puts the data in a circular buffer. The second thread removes data from the buffer and writes it to the SD.

You will need enough programming skill to modify these examples for your use.

Another approach is to use two arduinos. Use one for the data gathering and massaging, the second to do all the other work (i.e. writing to disk, etc.). I'm sampling power at 2.8ksps with a 328P @ 16MHz, the second chip (1284P) does the SPI work with the SD card as well as a RFM-12B. The two talk via serial, using EasyTransfer. I selected this approach to minimize the possibility of data getting lost while the 12B was transmitting, for example (it requires an interrupt).

A dedicated chip may also be an answer. For example, the specifications of a ADE7753 are tasty for power measurements, I just never got it to work properly. You might be able to find similar dedicated chips for sampling for your applications that can be addressed over I2C or SPI.

@fat16lib - Can I use your WaveRP library with this hardware?

http://www.ladyada.net/products/microsd/
I don't have the Wave Shield.
Where do I modify chipSelect like in the above tutorial?
Mine is not the default wiring.

I have built a mic preamp which works using analog input.
So I can record.
I could not play the files?
What do I need to change in the WaveRecordPlay example sketch?
What options can I change in that code?
What's the difference between SD and SDfat libraries?

The author says no changes are necessary. Can anyone verify this?
My chipSelect pin is not the default...