Fastest Data Logging? With SD Card?

You can't store into flash so that's out.

The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).


Rob

Maybe an idea
It may be an option not to write all data.
If the data is often the same as the previous data you may opt to write a repeat value instead of a new value.
So if your data is 1 1 2 2 2 2 2 3 3 3 3 4 4 4 4 you could write all data
1,1,2,2,2,2,2,3,3,3,3,4,4,4,4,5
or optimized data
12,25,34,44,5*1
Note that if you optimize this (never write *1) you will always save on writes.

However I am wondering that as you need this high frequency of reads whether repeats are expected. If not it is probably not worth the cpu cycles.
If you want this high frequency to notice spikes is it worth saving all the data?

As to the interrupt. Given the potential delays on SD cards a interrupt is your only guarantee for not missing reads.

Best regards
Jantje

Graynomad:
You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

http://arduino.cc/en/Reference/PROGMEM:
http://arduino.cc/en/Reference/PROGMEM:
Store data in flash (program) memory instead of SRAM.

Jantje:
It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

Graynomad:
The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).

I have yet to receive the final design specs, but it needs to be at least in the single digit milli second time.

I know that the Arduino is capable of microsecond time, and, in fact, according to the wisdom of the ancients, it is capable of at least 56KHz read speeds. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1208715493/15

On top of that, there used to be a library that could move data at incredibly fast speeds as well; but it has since 404'd...

But, there's another way using the i2c, increasing the read/write speed to 400KHz: http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1241668644/0

Hmmm.

I don't think that's true. What about with the prgspace library?

That's for storing read only data that you set at compile time - you can't write on progmem at runtime.

Neodudeman:

Graynomad:
You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

In my humble opinion you should rethink

Neodudeman:

Jantje:
It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

In my humble opinion you should rethink

But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

at least in the single digit milli second time.

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.


Rob

wildbill:
That's for storing read only data that you set at compile time - you can't write on progmem at runtime.

Ahh... Must have misread that one.

Graynomad:

But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

To the SD card..? Or am I really misunderstanding everything, in blind hope? :blush:

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.

Ok, so this idea does sound the best so far, but I think I'm unclear on the benefit of an interrupt..

  1. Set an interrupt that checks every 5ms
  2. Gather data in buffer A until 512
  3. Flag is set, and when interrupt check is called, it writes the data...?
  4. Data in buffer B is gathered, repeat...?

What makes an interrupt check better than having a looping check?
If I'm checking to see if buffer A is full to switch to buffer B, why is an interrupt better than if I were to simply write buffer A to the card in that same if block?

Or am I really misunderstanding everything, in blind hope?

Yep, I2C can't talk to an SD card. Normally you use SPI.

I think I'm unclear on the benefit of an interrupt..

simply write buffer A to the card in that same if block?

You could check in a loop and that may work, I'm not sure how the SD library operates. But if you call it to write 512 bytes and it blocks until the job is done you would miss some data. OTOH if the write is fast enough and/or the library doesn't block then you are OK. Hopefully someone who knows more about the SD library can help with that.

If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.


Rob

You may be using the wrong microcontroller for the task.

My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. at a similar hardware price and with pretty good IDEs as well. That should make sampling the amount of data you're envisioning a lot simpler than trying to make a Arduino do what it likely cannot (at least with the constraints you're envisioning).

But that's just me. FWIW, I recall reading that the smallest time slice a Arduino can offer is on the order of 65ns at 16MHz. That's for a single instruction. So, short, bursty signals may be better sampled on a higher-speed (i.e. SPI based) ADC that offers speed and resolution.

Constantin:
You may be using the wrong microcontroller for the task.
My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc.

Thanks for the suggestions; maybe going another route is a plausible solution, as long as the prices are similar.

Graynomad:
Yep, I2C can't talk to an SD card. Normally you use SPI.
If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

I see. So interrupts can possibly work concurrently? That's pretty great, then. RuggedCircuits' original solution will be the way to go.
Also, I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I think there is, but don't know where to get it.

This should be doable on the Ardiuno if the write speeds to the SD are fast enough.

RC said

On a good day SD cards can write a few hundred kB per second

You should be able to look at your requirements and do the maths based on that.

Let's be conservative and say 100kbps, at that speed you can write a 512-byte block in 5mS so we're in the ball park.


Rob

Here's an SD library by fat16lib

http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html

He's quoting nearly 200 KB/sec


Rob

..pls mind an sdcard can have ~250ms writing latency, so you need a buffer where you can store 250ms worth of data.. sdcard is not a linear memory you can push data at given rate into.. if you need a data logger for submicroseconds events you must know:

  1. what is the byte size of a single sample (ie adc=14bit = 2bytes)
  2. sample/event duration (ie a .2usec pulse)
  3. sampling/event's frequency (ie 1MHz)
  4. for how long you going to sample and store (ie 100days)
  5. etc.
    Then you may start to think on technology to be used. It is called a "detailed requirements specification".. :slight_smile:

This means you should have 3 buffers just in case when you hit this delay when you need to swap buffers.
Best regards
Jantje

For example: pic24/dspic33 (40-70MHz mcus) - you can sample at 1Msamples/sec (10bit, but let say 8bits to keep it simple), ADC is connected via DMA to two 2kB RAM blocks (buf1 and buf2) in so called ping-pong mode (ie you fill in the buf1 from the ADC and at the same time you process the buf2, and vice versa, all is done in hw automaticaly) - so you have 2msecs to process one buffer and to feed its content into buffer3 (ie via second DMA channel), which needs to contain worst case up to 250ms of data - that is 250kB of data. This could be ie an external SRAM, FRAM or MRAM (256kx8, serial/parallel) device. Then you need to move that buffer3 (or its actual content according to the watermark, ie via third DMA channel) into the sdcard (here you need a buffer4 - 512bytes or more for one or more sdcard's blocks). This will work when writing speed to the sdcard via SPI will be higher than 1MByte/sec (to empty the buffer3 fast - here probably an SDIO interface would be a better choice).
So this is maybe a "feasible" situation. If you are going to store into an external RAM device, ie serial/parallel SRAM, you do not need the buffer3..
Not doable with arduino, however. p.

Write latency is the big problem with SD cards. SD cards are designed to do large contiguous writes. Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated. This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands. This reduces the maximum write latency. I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency. 8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at Google Code Archive - Long-term storage for Google Code Project Hosting..

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384.

@fat16lib: what is the typical wr latency with that sdfat feature, pls?

There is not really a typical write latency. It depends on the card design, its history of use, how it was formatted, how fast you are trying to write in blocks/second.

I wrote a test program, SdFatRawWrite. It is included with SdFat in the extra/examplesV1/SdFatRawWrite folder. SdFat has had raw write for a long time.

I like SanDisk cards, 1 GB or 2 GB formatted FAT16.

I ran SdFatRawWrite on an old 1 GB Sandisk Ultra II card formatted FAT16. I wrote a 10 MB file at 102,400 bytes per second. The result was:

Start raw write of 10240000 bytes at
102400 bytes per second
Please wait 100 seconds
Done
Overruns: 0
Elapsed time: 100000 millis
Max write time: 828 micros

This card has extremely good performance. The max time to write a 512 byte block was 828 usec.

This card was manufactured in 12/2006. Many newer cards can't beat it.

I see.. We've spent a lot of time elaborating the sdcard stuff (see retrobsd.org ) as we use the sdcard for bsd unix filesystem and swap file as well. With pic32 we have achieved 1.7MBytes/sec read (SPI, no DMA) and ~800kB/sec write afaik, but with time (the swap swaps about 200kB/sec data based on number of jobs running) some of my cards started to become very slow after few weeks. The fastest cards are 256MB and 512MB ones, but hard to find today..

Once again, write speed is the wrong measure for data logging with limited RAM. BSD swap and file system performance is irrelevant.

The limit is the amount of buffer memory and the maximum write latency for a block. Buffer size divided by max latency is the speed limit.

The write speed spec for high speed SDHC cards is based on writing many MB of data. This is reasonable for video cameras that have huge buffers. These cards may have poor performance in Arduino data logging since so little RAM is available to handle the occasional long write latency these cards have.

Edit:

The above card has a very low write latency as long as you allow at least four milliseconds between writes. If you push it to one block every three milliseconds it will have an occasional 14-15 millisecond latency. So the max data logging rate has little to do with the max write rate. I assume you don't want to miss data points.

If you write this card at max speed, the rate is 428 KB/sec but there are now occasional 44 millisecond write latencies.