Fastest Data Logging? With SD Card?

You may be using the wrong microcontroller for the task.

My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. at a similar hardware price and with pretty good IDEs as well. That should make sampling the amount of data you're envisioning a lot simpler than trying to make a Arduino do what it likely cannot (at least with the constraints you're envisioning).

But that's just me. FWIW, I recall reading that the smallest time slice a Arduino can offer is on the order of 65ns at 16MHz. That's for a single instruction. So, short, bursty signals may be better sampled on a higher-speed (i.e. SPI based) ADC that offers speed and resolution.

Constantin:
You may be using the wrong microcontroller for the task.
My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc.

Thanks for the suggestions; maybe going another route is a plausible solution, as long as the prices are similar.

Graynomad:
Yep, I2C can't talk to an SD card. Normally you use SPI.
If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

I see. So interrupts can possibly work concurrently? That's pretty great, then. RuggedCircuits' original solution will be the way to go.
Also, I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I think there is, but don't know where to get it.

This should be doable on the Ardiuno if the write speeds to the SD are fast enough.

RC said

On a good day SD cards can write a few hundred kB per second

You should be able to look at your requirements and do the maths based on that.

Let's be conservative and say 100kbps, at that speed you can write a 512-byte block in 5mS so we're in the ball park.


Rob

Here's an SD library by fat16lib

http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html

He's quoting nearly 200 KB/sec


Rob

..pls mind an sdcard can have ~250ms writing latency, so you need a buffer where you can store 250ms worth of data.. sdcard is not a linear memory you can push data at given rate into.. if you need a data logger for submicroseconds events you must know:

  1. what is the byte size of a single sample (ie adc=14bit = 2bytes)
  2. sample/event duration (ie a .2usec pulse)
  3. sampling/event's frequency (ie 1MHz)
  4. for how long you going to sample and store (ie 100days)
  5. etc.
    Then you may start to think on technology to be used. It is called a "detailed requirements specification".. :slight_smile:

This means you should have 3 buffers just in case when you hit this delay when you need to swap buffers.
Best regards
Jantje

For example: pic24/dspic33 (40-70MHz mcus) - you can sample at 1Msamples/sec (10bit, but let say 8bits to keep it simple), ADC is connected via DMA to two 2kB RAM blocks (buf1 and buf2) in so called ping-pong mode (ie you fill in the buf1 from the ADC and at the same time you process the buf2, and vice versa, all is done in hw automaticaly) - so you have 2msecs to process one buffer and to feed its content into buffer3 (ie via second DMA channel), which needs to contain worst case up to 250ms of data - that is 250kB of data. This could be ie an external SRAM, FRAM or MRAM (256kx8, serial/parallel) device. Then you need to move that buffer3 (or its actual content according to the watermark, ie via third DMA channel) into the sdcard (here you need a buffer4 - 512bytes or more for one or more sdcard's blocks). This will work when writing speed to the sdcard via SPI will be higher than 1MByte/sec (to empty the buffer3 fast - here probably an SDIO interface would be a better choice).
So this is maybe a "feasible" situation. If you are going to store into an external RAM device, ie serial/parallel SRAM, you do not need the buffer3..
Not doable with arduino, however. p.

Write latency is the big problem with SD cards. SD cards are designed to do large contiguous writes. Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated. This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands. This reduces the maximum write latency. I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency. 8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at Google Code Archive - Long-term storage for Google Code Project Hosting..

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384.

@fat16lib: what is the typical wr latency with that sdfat feature, pls?

There is not really a typical write latency. It depends on the card design, its history of use, how it was formatted, how fast you are trying to write in blocks/second.

I wrote a test program, SdFatRawWrite. It is included with SdFat in the extra/examplesV1/SdFatRawWrite folder. SdFat has had raw write for a long time.

I like SanDisk cards, 1 GB or 2 GB formatted FAT16.

I ran SdFatRawWrite on an old 1 GB Sandisk Ultra II card formatted FAT16. I wrote a 10 MB file at 102,400 bytes per second. The result was:

Start raw write of 10240000 bytes at
102400 bytes per second
Please wait 100 seconds
Done
Overruns: 0
Elapsed time: 100000 millis
Max write time: 828 micros

This card has extremely good performance. The max time to write a 512 byte block was 828 usec.

This card was manufactured in 12/2006. Many newer cards can't beat it.

I see.. We've spent a lot of time elaborating the sdcard stuff (see retrobsd.org ) as we use the sdcard for bsd unix filesystem and swap file as well. With pic32 we have achieved 1.7MBytes/sec read (SPI, no DMA) and ~800kB/sec write afaik, but with time (the swap swaps about 200kB/sec data based on number of jobs running) some of my cards started to become very slow after few weeks. The fastest cards are 256MB and 512MB ones, but hard to find today..

Once again, write speed is the wrong measure for data logging with limited RAM. BSD swap and file system performance is irrelevant.

The limit is the amount of buffer memory and the maximum write latency for a block. Buffer size divided by max latency is the speed limit.

The write speed spec for high speed SDHC cards is based on writing many MB of data. This is reasonable for video cameras that have huge buffers. These cards may have poor performance in Arduino data logging since so little RAM is available to handle the occasional long write latency these cards have.

Edit:

The above card has a very low write latency as long as you allow at least four milliseconds between writes. If you push it to one block every three milliseconds it will have an occasional 14-15 millisecond latency. So the max data logging rate has little to do with the max write rate. I assume you don't want to miss data points.

If you write this card at max speed, the rate is 428 KB/sec but there are now occasional 44 millisecond write latencies.

fat16lib:
Write latency is the big problem with SD cards. SD cards are designed to do large contiguous writes. Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated. This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands. This reduces the maximum write latency. I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency. 8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at Google Code Archive - Long-term storage for Google Code Project Hosting..

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384.

Thank you, everyone! Your help is greatly appreciated. I've taken all your advice, and will ponder on it for a while.

And, a special thanks to you, Fat16Lib.
Because of hard working, generous gurus like yourself, arduino-laymen like myself can utilize the wisdom of the masters!

Hi Guys,

Only new to the Arduino world so please take easy on me...

I'm having issues writing data to a 1 GB SD card. I have formatted it to FAT 16. I'm using a Mega ADK with the WiFi shield to log data to the SD.

It a very basic setup... trim-pot resistor connected to A0. All I want to do is read and log the voltage to a .csv file - voltage needs to be a float but have tried it as an integer with no increase in logging speed.

The speed of logging the data to the SD is very very slow - about 4 s per sample. For what I have read the limitation of the SD card is in the order of ms not s. I was thinking is has something to do with the buffer becoming full but for what I have read the .close() function should clear the buffer.

Can somebody please tell me why my code is so slow or load it for themselves and see if they experience the same issues?

Any help would be much appreciated.

Cheers

Adam

majorProject.ino (1.94 KB)

I have looked at your code and can't see what you are doing with strings, hopefully nothing.

You may find the SD is fine but your hardware is struggling with the serial rate. Since you have a 1 sec delay in the loop there can't be a very good reason for not using the default 9600.

I use pin 10, output for both Uno and plain vanilla Mega but I guess if this was a problem it wouldn't work at all and what you have is kosher for the ADK Mega.

I don't understand the

  }
  else if(id>10)
  {
    return;
  }
  else
  {
    Serial.println("Could not open file...");
    return;
  }

But, if you eventually get what you want, I guess it's kosher.

Hi Nick,

Thanks for your comments.

Regarding the bit of code that didn't makes sense to you... just some error checking while the program is still in the development phase. I wouldn't have the program stop until there was an input from the user for the real version.

In the current form the program only collects 10 samples. I can manage to collect and log up to 32 samples quickly i.e. order of ms to csv however; anymore samples seems to make the program hit a brick wall. Sample rate decreases dramatically and csv file becomes corrupted.

The else statement just see if the file can be opened and if it can't displays the error to the user.

I have tested the program with and without the delay(1000) and it doesn't seem to have any effect of the sample rate after > 30 samples.

I have also changed the baud rate back to 9600 and still not luck getting the thing to work effectively.

What's a "kosher"?

As far as I know there is little difference between the normal and ADK Mega boards in regards to their pin mapping. The following link states that pin 53 (SPI SS pin) on the Mega must be set as an OUTPUT - not pin 10. So I don't see how your code can work on the Mega.

I only use the strings for serials comms and to write headers at the tops of the columns in the csv file. No string are used during the data collection and logging to csv.

Any other ideas mate?

I'm starting to run short of them quite quickly. I believe I have been living in something of a fools' paradise and your post is confirming it. Last year I was obliged to move from Uno to Mega and sorted out the clock and analogue pin calls. You are right in that 53 is the SPI SS pin, and 10>13 do not carry the SPI. Nonetheless, by carrying the code over from the Uno, I have pinmode 10,output and it has been working just fine for months.

I'm not sure I have an explanation for this. It could be because I meticulously keep the SD operations tightly bound together, to the exclusion of all else.

SD.open tralala
dataFile.print......
dataFile.print......
dataFile.close

I don't recall a technical reason for doing this, or even when I started doing it, it just looks good on paper. Having the same tight operation can't possibly be a bad idea, but I wouldn't recommend calling pin 10 to anybody.

I deleted your two string commands and the file compiles OK, which is just as well. It seems that anything to do with data strings is either poisonous, a waste of time, or both. More to the point, I now understand that your programme works fine initially and then hits a brick wall after 32 samples. This might indicate that you are running out of memory. I have never experienced this, principally because I never accumulate data, it just passes through on its way to SD, LCD, Bluetooth, and the internet. Problem is, your code isn't very different. The only thing I can see that I clearly wouldn't do is the line

long id=1

indeed I would be very suss about it. The way I see it, it means you could count seconds up to 2,147,483,648 i.e. for 68 years, before the variable is exhausted. There are likely to be memory implications here and you might try using a simple

int id=1

Hi Nick,

I have managed to get the code going great now. There was an error in the code for using the WiFi sheild as a SD card reader/logger only.

I did a little more reading and found out that not only do you need pin 53 set as an OUTPUT but if you want to use the SD function of the shield explicitly used must deselect the WiFi chip (HDG104). This is done by setting pin 10 as an HIGH OUPUT. Apparently if I imported the WiFI library I would need to explicitly deselect the chip.

// libraries used
#include <SD.h>

// declare global constants and their data types
const int trimpotSensor=A0; // variable resistor on analog pin 0
const int chipSelect=4; // Chip Select pin default channel on EtherMega board for SD communication

// declare global variables and their data types
int sensorValue;
int id=1;
float volts;

// declare files
File dataFile;

// setup code here, to run once
void setup() 
{
  Serial.begin(9600); // initialize serial communication
  while(!Serial)
  {
  }
  Serial.print("Initializing SD card..."); // print to the serial monitor a message
  pinMode(53,OUTPUT); // pin 53 must be left as an OUTPUT for the SD library to work
  pinMode(10,OUTPUT); // pin 10 must be left as an OUTPUT so it can be set HIGH
  digitalWrite(10,HIGH); // pin 10 needs to be put HIGH to explicitly deselect the WiFi HDG104 chip
  if(!SD.begin(chipSelect)) // if the statement !SD.begin(SS) comes back as FALSE execute the "if" statement
  {
    Serial.println("Card failed or not present"); // print to screen the SD card status
    return; // start the loop again and check if the SD is faulty or not installed
  }
  else
  {
    Serial.println("Card initialized"); // else everything is all good
  }
  if(SD.exists("volts.csv"))
  {
    Serial.println("File exists and now being removed");
    SD.remove("volts.csv");
  }
  else
  {
    Serial.println("No file could be found");
  }
  Serial.print("Sample No.");
  Serial.print(",");
  Serial.println("Volts");
  dataFile=SD.open("volts.csv",FILE_WRITE);
  dataFile.print("Sample No.");
  dataFile.print(",");
  dataFile.println("Volts");
  dataFile.close();
}

// main code
void loop() 
{
  sensorValue=analogRead(trimpotSensor);
  volts=5.0/1024.0*float(sensorValue);
  dataFile=SD.open("volts.csv",FILE_WRITE);
  if(dataFile && id<=50)
  {
    delay(150);
    dataFile.print(id,DEC);
    dataFile.print(",");
    dataFile.println(volts,DEC);
    dataFile.close();
    Serial.print(id,DEC);
    Serial.print(",");
    Serial.println(volts,DEC);
  }
  else
  {
    return;
  }
  id++;
}

Also found out the SD card I was using was pretty crappy. Got myself a good 8 GB ScanDisk SD and it can log data in the order of ms now. So all is good for now.

Thanks a lot for your assistance. Also changed the counter to an int variable so that could also be a factor.

Cheers

Adam

Hello guys! I know this post is somewhat old but is very much related to my question:

I have to write to an SD card 307,200 bytes at 1 byte per microsecond (max). I'm testing the RawWrite example from SdFat library setting 512 µS per block (512 bytes) but the best I see in the log is 708 µS per block. In fact, the total time is more than twice the expected time, so I guess the average write is above 1024 µS. The max write time is 35 mS which is far more than my application limit.

I have to dump 307,200 bytes from a AL422B FIFO to the SD. This chip supports a minimum reading clock of 1MHz.

Based in the numbers I've seen in this post and the few tests I've made, there is no way I can do this. But just in case, is there anyone there who got better results?

Which sd card are you using.