next topic »

[on:]

Hi,

I am working on a project in which I need to sample at 1KHz and be able to read, write, and transmit data over serial bluetooth in between samples.  I'm writing the data to an SD card using the SD lib in arduino 0022.  Using the O_WRITE and O_OPEN commands from the library that SD.h wraps I am able to get write times of ~400-550 microseconds for 2 byte integers.  That doesn't leave me enough time to read and transmit when I would need to.  The SPI clock is running at 8MHz so I should be able to write much faster that this, but I am too inexperienced in C/C++ to sift through the code and decipher where the fat (pun intended) needs trimming.

If anyone can help it would be much appreciated.

[Reply #1 on:]

You will never achieve reliable data logging at 1 ksps with the SD.h wrapper.  It is possible to log data at high rates with SdFat but it requires special techniques.  The problem is latency and buffering, not the SPI bus.

I wrote the WaveRP library to record audio at up to 44.1 ksps for 8-bit data.  It is based on SdFat but is complex.  See the new beta here http://code.google.com/p/waverp/downloads/list

[Reply #2 on:]

It is possible to log data at high rates with SdFat but it requires special techniques.  The problem is latency and buffering, not the SPI bus.

What special techniques would I need to implement to get a higher read/write times?  Do I need to change the buffer size?

[Reply #3 on:]

Here is one trick I read from another post here:

Use multiple EEPROMS. EEPROMs require much less CPU for access so it's faster. When you gang them up, then write to ROM1, then to ROM2, then ROM3... you save time since you don't have to wait till ROM1 finishes writing. You have four of them together (very easy to wire, no surface mount needed), do your measurements, write to ROM1, do measurements again, write to ROM2, so on. It may help speed up your process quite a bit when all you have to do is transferring data and the ROM does the writing. Using sd cards is quite cpu intensive.

[Reply #4 on:]

Sd.h should be able to handle 1ks/s quite easily. In fact, I'm currently doing 2ks/s + anothre 500/s via xbee, and dumping it all to an SD card in real time.

All you really need to do is create a buffer. Writing 2 bytes at a time is EXTREMELY slow.

Code:

void * temp =malloc(BUFFER_SIZE * sizeof (unsigned short));
if(temp != null)
  sampleBuffer = (unsigned short*) temp;
else{
  //Out of memory
}

fill your buffer with samples, then write the entire thing at once. Writing a buffer of 100 Bytes doesn't take much longer then writing a single value.

[Reply #5 on:]

You can write at an average rate of well over 2 KB per second with a buffer but this is not 1 ksps with two byte samples.  You will miss samples unless you capture data in an IRS or using a device with buffering like the xbee.

The reason is that SdFat which SD.h is bases on will take far more than a millisecond to do a write when a new cluster is allocated to the file.

SD cards are designed for sequential writes so accessing the two copies of the file allocation table causes long latency.  20 - 30 ms is typical for allocating a cluster.

To record audio with no missed samples I use two buffers and and capture data in a timer driven IRS.

[Reply #6 on:]

Does the SD card class make a difference or is it the hardware most Arduino users use?  Could we get full speed from a class 10 card?

[Reply #7 on:]

I'm still a little unsure on how to implement the buffers.  Currently I have an output compare interrupt on the 16 bit timer that allows me to sample at 1kHz and I am just writing the ADC data to the SD card every time the ISR fires.  To implement the buffer system you described, do I just write to one buffer and when it fills write it to the SD card while the sampling continues to fill up a second buffer then just alternate like that?  I'm also unsure how to write an entire buffer at once.

[Reply #8 on:]

The card class is not important.  You need very large multiple block writes to get the advantage of class 10 cards.

Yes, fill one buffer in the interrupt routine while the other is being written.  For best results ues buffers with a size of 128, 256, or 512 bytes.  This works best since a buffer will align on SD block boundries.

[Reply #9 on:]

I think I'm thinking about writing the buffers to the SD card incorrectly... It can't be that I write each value incrementally, that would be the same thing as writing each as it came in from the ADC, right?  I've been able to get write times to ~190 microseconds  (~10kB/sec) using:

Code:

char buff[4];
sprintf(buff, "%i",<value from adc>);
myFile.write(buff);
myFile.write("\r\n");

This is significantly faster than the println route.  Can you help me on how to write an entire buffer at once, without blocking the ISR.  It is critical that I do not lose samples. 

[Reply #10 on:]

Maybe the eaisest way to avoid missed samples is to use a ring buffer like Serial.  Put the binary ADC values in the buffer in the ISR and remove and format the values in the background.

Using sprintf is faster than Arduino print.  I suggested an improved smaller version of print that is faster and smaller but I guess the Arduino group has no intrest.

You should use the format string "%i\r\n" so you can use one write.

Look at the Arduino hardware serial driver for details on using the ring buffer.

[Reply #11 on:]

Hey, I was just testing using the "%i\r\n in sprintf and I noticed that it actually takes shorter to write when that is done than when you break it up into 2 writes like I had originally posted it.  By shorter I mean 5 microseconds, but the bigger problem I ran into with sprintf(buff,"%i\r\n",analogRead(0)) was that it often inserted characters in between readings instead of new lines.  I'm also reading the values off to an android phone through bluetooth and this really threw a wrench in the software I had quickly written up to read the values out.  Do you have any idea why it would be doing this?  I tried echoing out the data being written to the SD through the UART and found the same errors so it wasn't an error in the writing to the SD. 

[Reply #12 on:]

but the bigger problem I ran into with sprintf(buff,"%i\r\n",analogRead(0)) was that it often inserted characters in between readings instead of new lines.

Sounds to me like buff isn't big enough. The %i format statement will require up to 4 characters to hold the 1023 maximum value returned by analogRead. The \n, \r, and trailing NULL add three more, so buff needs to be at least 7 characters long. How big is it?

[Reply #13 on:]

You're exactly right.  I had the char buffer at 4 when I was just writing the 4 characters from the ADC.  I forgot to add some more for the escape characters. 

[Reply #14 on:]

Hi,
I have 3 sensors attached to a n Arduino board with atmega 328 that I am using for a motion capture project. The sensors are and ADX345 accelerometer, HMC 5883L magnetometer and IDG3200 gyro. I have the sensors working correctly and I can sample them just fine. The problem I have is writing to the micro SD. I'd like to be able to write the data at a rate of about 200Hz. Does anyone have he code to do this?