Fastest Data Logging? With SD Card?

Hello! I've got a question that I hope an arduino guru can help me out with.

I want to build a system where the arduino does some high resolution data logging of analog inputs. For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.

I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?
Is this really the best way? What other ways can this be done?

I stumbled upon this: http://arduino.cc/forum/index.php/topic,77378.0.html
But what's the best way without changing the kernel?

Thanks in advance!

For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.

Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).

I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?

It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.

--
The Gadget Shield: accelerometer, RGB LED, IR transmit/receive, speaker, microphone, light sensor, potentiometer, pushbuttons

RuggedCircuits:
Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).

Right, yes, sorry; got my ns mixed with my us. I meant to write microseconds, but I guess that's still impossible anyway.

RuggedCircuits:
It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.

Great, thanks for that info. I'm worried, though, that interrupts would mess with the timing of the code; and logging the data with high resolution, accurate, time is a key specification. Actually, what about storing to the Flash memory? Would that slow anything down, do you think?

You can't store into flash so that's out.

The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).


Rob

Maybe an idea
It may be an option not to write all data.
If the data is often the same as the previous data you may opt to write a repeat value instead of a new value.
So if your data is 1 1 2 2 2 2 2 3 3 3 3 4 4 4 4 you could write all data
1,1,2,2,2,2,2,3,3,3,3,4,4,4,4,5
or optimized data
12,25,34,44,5*1
Note that if you optimize this (never write *1) you will always save on writes.

However I am wondering that as you need this high frequency of reads whether repeats are expected. If not it is probably not worth the cpu cycles.
If you want this high frequency to notice spikes is it worth saving all the data?

As to the interrupt. Given the potential delays on SD cards a interrupt is your only guarantee for not missing reads.

Best regards
Jantje

Graynomad:
You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

http://arduino.cc/en/Reference/PROGMEM:
http://arduino.cc/en/Reference/PROGMEM:
Store data in flash (program) memory instead of SRAM.

Jantje:
It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

Graynomad:
The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).

I have yet to receive the final design specs, but it needs to be at least in the single digit milli second time.

I know that the Arduino is capable of microsecond time, and, in fact, according to the wisdom of the ancients, it is capable of at least 56KHz read speeds. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1208715493/15

On top of that, there used to be a library that could move data at incredibly fast speeds as well; but it has since 404'd...

But, there's another way using the i2c, increasing the read/write speed to 400KHz: http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1241668644/0

Hmmm.

I don't think that's true. What about with the prgspace library?

That's for storing read only data that you set at compile time - you can't write on progmem at runtime.

Neodudeman:

Graynomad:
You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

In my humble opinion you should rethink

Neodudeman:

Jantje:
It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

In my humble opinion you should rethink

But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

at least in the single digit milli second time.

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.


Rob

wildbill:
That's for storing read only data that you set at compile time - you can't write on progmem at runtime.

Ahh... Must have misread that one.

Graynomad:

But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

To the SD card..? Or am I really misunderstanding everything, in blind hope? :blush:

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.

Ok, so this idea does sound the best so far, but I think I'm unclear on the benefit of an interrupt..

  1. Set an interrupt that checks every 5ms
  2. Gather data in buffer A until 512
  3. Flag is set, and when interrupt check is called, it writes the data...?
  4. Data in buffer B is gathered, repeat...?

What makes an interrupt check better than having a looping check?
If I'm checking to see if buffer A is full to switch to buffer B, why is an interrupt better than if I were to simply write buffer A to the card in that same if block?

Or am I really misunderstanding everything, in blind hope?

Yep, I2C can't talk to an SD card. Normally you use SPI.

I think I'm unclear on the benefit of an interrupt..

simply write buffer A to the card in that same if block?

You could check in a loop and that may work, I'm not sure how the SD library operates. But if you call it to write 512 bytes and it blocks until the job is done you would miss some data. OTOH if the write is fast enough and/or the library doesn't block then you are OK. Hopefully someone who knows more about the SD library can help with that.

If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.


Rob

You may be using the wrong microcontroller for the task.

My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. at a similar hardware price and with pretty good IDEs as well. That should make sampling the amount of data you're envisioning a lot simpler than trying to make a Arduino do what it likely cannot (at least with the constraints you're envisioning).

But that's just me. FWIW, I recall reading that the smallest time slice a Arduino can offer is on the order of 65ns at 16MHz. That's for a single instruction. So, short, bursty signals may be better sampled on a higher-speed (i.e. SPI based) ADC that offers speed and resolution.

Constantin:
You may be using the wrong microcontroller for the task.
My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc.

Thanks for the suggestions; maybe going another route is a plausible solution, as long as the prices are similar.

Graynomad:
Yep, I2C can't talk to an SD card. Normally you use SPI.
If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

I see. So interrupts can possibly work concurrently? That's pretty great, then. RuggedCircuits' original solution will be the way to go.
Also, I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I think there is, but don't know where to get it.

This should be doable on the Ardiuno if the write speeds to the SD are fast enough.

RC said

On a good day SD cards can write a few hundred kB per second

You should be able to look at your requirements and do the maths based on that.

Let's be conservative and say 100kbps, at that speed you can write a 512-byte block in 5mS so we're in the ball park.


Rob

Here's an SD library by fat16lib

http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html

He's quoting nearly 200 KB/sec


Rob

..pls mind an sdcard can have ~250ms writing latency, so you need a buffer where you can store 250ms worth of data.. sdcard is not a linear memory you can push data at given rate into.. if you need a data logger for submicroseconds events you must know:

  1. what is the byte size of a single sample (ie adc=14bit = 2bytes)
  2. sample/event duration (ie a .2usec pulse)
  3. sampling/event's frequency (ie 1MHz)
  4. for how long you going to sample and store (ie 100days)
  5. etc.
    Then you may start to think on technology to be used. It is called a "detailed requirements specification".. :slight_smile:

This means you should have 3 buffers just in case when you hit this delay when you need to swap buffers.
Best regards
Jantje

For example: pic24/dspic33 (40-70MHz mcus) - you can sample at 1Msamples/sec (10bit, but let say 8bits to keep it simple), ADC is connected via DMA to two 2kB RAM blocks (buf1 and buf2) in so called ping-pong mode (ie you fill in the buf1 from the ADC and at the same time you process the buf2, and vice versa, all is done in hw automaticaly) - so you have 2msecs to process one buffer and to feed its content into buffer3 (ie via second DMA channel), which needs to contain worst case up to 250ms of data - that is 250kB of data. This could be ie an external SRAM, FRAM or MRAM (256kx8, serial/parallel) device. Then you need to move that buffer3 (or its actual content according to the watermark, ie via third DMA channel) into the sdcard (here you need a buffer4 - 512bytes or more for one or more sdcard's blocks). This will work when writing speed to the sdcard via SPI will be higher than 1MByte/sec (to empty the buffer3 fast - here probably an SDIO interface would be a better choice).
So this is maybe a "feasible" situation. If you are going to store into an external RAM device, ie serial/parallel SRAM, you do not need the buffer3..
Not doable with arduino, however. p.

Write latency is the big problem with SD cards. SD cards are designed to do large contiguous writes. Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated. This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands. This reduces the maximum write latency. I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency. 8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at Google Code Archive - Long-term storage for Google Code Project Hosting..

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384.

@fat16lib: what is the typical wr latency with that sdfat feature, pls?