Pages: [1] 2 3   Go Down
Author Topic: Fastest Data Logging? With SD Card?  (Read 15001 times)
0 Members and 1 Guest are viewing this topic.
0
Offline Offline
Newbie
*
Karma: 0
Posts: 29
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hello! I've got a question that I hope an arduino guru can help me out with.

I want to build a system where the arduino does some high resolution data logging of analog inputs. For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.

I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?
Is this really the best way? What other ways can this be done?

I stumbled upon this: http://arduino.cc/forum/index.php/topic,77378.0.html
But what's the best way without changing the kernel?

Thanks in advance!
Logged

0
Offline Offline
Faraday Member
**
Karma: 13
Posts: 2857
ruggedcircuits.com
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.

Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).

Quote
I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?

It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.

--
The Gadget Shield: accelerometer, RGB LED, IR transmit/receive, speaker, microphone, light sensor, potentiometer, pushbuttons
Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 29
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset


Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).

Right, yes, sorry; got my ns mixed with my us. I meant to write microseconds, but I guess that's still impossible anyway.

It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.

Great, thanks for that info. I'm worried, though, that interrupts would mess with the timing of the code; and logging the data with high resolution, accurate, time is a key specification. Actually, what about storing to the Flash memory? Would that slow anything down, do you think?
Logged

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8445
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

You can't store into flash so that's out.

The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).

_______
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

Belgium
Offline Offline
Edison Member
*
Karma: 58
Posts: 1731
Arduino rocks; but with my plugin it can fly rocking the world ;-)
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Maybe an idea
It may be an option not to write all data.
If the data is often the same as the previous data you may opt to write a repeat value instead of a new value.
So if your data is 1 1 2 2 2 2 2 3 3 3 3 4 4 4 4 you could write all data
1,1,2,2,2,2,2,3,3,3,3,4,4,4,4,5
or optimized data
1*2,2*5,3*4,4*4,5*1
Note that if you optimize this (never write *1) you will always save on writes.

However I am wondering that as you need this high frequency of reads whether repeats are expected. If not it is probably not worth the cpu cycles.
If you want this high frequency to notice spikes is it worth saving all the data?

As to the interrupt. Given the potential delays on SD cards a interrupt is your only guarantee for not missing reads.

Best regards
Jantje
Logged

Do not PM me a question unless you are prepared to pay for consultancy.
Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -

0
Offline Offline
Newbie
*
Karma: 0
Posts: 29
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

You can't store into flash so that's out.
I don't think that's true. What about with the prgspace library?

http://arduino.cc/en/Reference/PROGMEM:
Store data in flash (program) memory instead of SRAM.

It may be an option not to write all data.
Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).
I have yet to receive the final design specs, but it needs to be at least  in the single digit milli second time.

I know that the Arduino is capable of microsecond time, and, in fact, according to the wisdom of the ancients, it is capable of at least 56KHz read speeds. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1208715493/15

On top of that, there used to be a library that could move data at incredibly fast speeds as well; but it has since 404'd...
http://arduinonut.blogspot.com/2009/02/libraries.html

But, there's another way using the i2c, increasing the read/write speed to 400KHz: http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1241668644/0

Hmmm.
Logged

New Jersey
Offline Offline
Faraday Member
**
Karma: 48
Posts: 3417
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I don't think that's true. What about with the prgspace library?

That's for storing read only data that you set at compile time - you can't write on progmem at runtime.
Logged

Belgium
Offline Offline
Edison Member
*
Karma: 58
Posts: 1731
Arduino rocks; but with my plugin it can fly rocking the world ;-)
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

You can't store into flash so that's out.
I don't think that's true. What about with the prgspace library?
In my humble opinion you should rethink

It may be an option not to write all data.
Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.
In my humble opinion you should rethink
Logged

Do not PM me a question unless you are prepared to pay for consultancy.
Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8445
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
But, there's another way using the i2c, increasing the read/write speed to 400KHz:
But where will you move it to?

Quote
at least  in the single digit milli second time.
1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.

______
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

0
Offline Offline
Newbie
*
Karma: 0
Posts: 29
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

That's for storing read only data that you set at compile time - you can't write on progmem at runtime.

Ahh... Must have misread that one.

Quote
But, there's another way using the i2c, increasing the read/write speed to 400KHz:
But where will you move it to?
To the SD card..? Or am I really misunderstanding everything, in blind hope?  smiley-red

Quote
1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.

Ok, so this idea does sound the best so far, but I think I'm unclear on the benefit of an interrupt..

1. Set an interrupt that checks every 5ms
2. Gather data in buffer A until 512
3. Flag is set, and when interrupt check is called, it writes the data...?
4. Data in buffer B is gathered, repeat...?

What makes an interrupt check better than having a looping check?
If I'm checking to see if buffer A is full to switch to buffer B, why is an interrupt better than if I were to simply write buffer A to the card in that same if block?
Logged

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8445
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Or am I really misunderstanding everything, in blind hope?
Yep, I2C can't talk to an SD card. Normally you use SPI.

Quote
I think I'm unclear on the benefit of an interrupt..
Quote
simply write buffer A to the card in that same if block?
You could check in a loop and that may work, I'm not sure how the SD library operates. But if you call it to write 512 bytes and it blocks until the job is done you would miss some data. OTOH if the write is fast enough and/or the library doesn't block then you are OK. Hopefully someone who knows more about the SD library can help with that.

If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

______
Rob
« Last Edit: March 30, 2012, 07:38:19 am by Graynomad » Logged

Rob Gray aka the GRAYnomad www.robgray.com

Peoples Republic of Cantabrigia
Offline Offline
God Member
*****
Karma: 6
Posts: 691
Arduino happiness
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

You may be using the wrong microcontroller for the task.

My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. at a similar hardware price and with pretty good IDEs as well. That should make sampling the amount of data you're envisioning a lot simpler than trying to make a Arduino do what it likely cannot (at least with the constraints you're envisioning).

But that's just me. FWIW, I recall reading that the smallest time slice a Arduino can offer is on the order of 65ns at 16MHz. That's for a single instruction. So, short, bursty signals may be better sampled on a higher-speed (i.e. SPI based) ADC that offers speed and resolution.
« Last Edit: March 30, 2012, 08:56:53 am by Constantin » Logged

0
Offline Offline
Newbie
*
Karma: 0
Posts: 29
Arduino rocks
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset



You may be using the wrong microcontroller for the task.
My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. 

Thanks for the suggestions; maybe going another route is a plausible solution, as long as the prices are similar.


Yep, I2C can't talk to an SD card. Normally you use SPI.
If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.
I see. So interrupts can possibly work concurrently? That's pretty great, then. RuggedCircuits' original solution will be the way to go.
Also, I've heard there's a faster alternative to the standard arduino library that I can use; is this true?
Logged

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8445
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I've heard there's a faster alternative to the standard arduino library that I can use; is this true?
I think there is, but don't know where to get it.

This should be doable on the Ardiuno if the write speeds to the SD are fast enough.

RC said

Quote
On a good day SD cards can write a few hundred kB per second

You should be able to look at your requirements and do the maths based on that.

Let's be conservative and say 100kbps, at that speed you can write a 512-byte block in 5mS so we're in the ball park.

_______
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

nr Bundaberg, Australia
Offline Offline
Tesla Member
***
Karma: 121
Posts: 8445
Scattered showers my arse -- Noah, 2348BC.
View Profile
WWW
 Bigger Bigger  Smaller Smaller  Reset Reset

Here's an SD library by fat16lib

http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html

He's quoting nearly 200 KB/sec

_____
Rob
Logged

Rob Gray aka the GRAYnomad www.robgray.com

Pages: [1] 2 3   Go Up
Jump to: