Arduino Forum

Topics => Science and Measurement => Topic started by: Neodudeman on Mar 29, 2012, 04:21 am

Title: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Mar 29, 2012, 04:21 am
Hello! I've got a question that I hope an arduino guru can help me out with.

I want to build a system where the arduino does some high resolution data logging of analog inputs. For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.

I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?
Is this really the best way? What other ways can this be done?

I stumbled upon this: http://arduino.cc/forum/index.php/topic,77378.0.html
But what's the best way without changing the kernel?

Thanks in advance!
Title: Re: Fastest Data Logging? With SD Card?
Post by: RuggedCircuits on Mar 29, 2012, 04:46 am
Quote
For example, the voltage on an analog pin. I'd like it to be able to log in millisecond time (or nano second, considering how fast one can read/write with the ports with port registers), and then save it to an SD card.


Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).

Quote
I'm thinking the best way to do it is to save the data to the arduino's internal memory, for the sake of speed, and once that is filled, write it to the SD card. However, I think the limit is the size of the arduino's internal memory; how much data can an UNO really handle?


It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.

--
The Gadget Shield (http://ruggedcircuits.com/html/gadget_shield.html): accelerometer, RGB LED, IR transmit/receive, speaker, microphone, light sensor, potentiometer, pushbuttons
Title: Re: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Mar 29, 2012, 09:38 am


Whoa there, let's make sure we're on the same page. Millisecond log times are entirely doable with an SD card, nanoseconds are entirely impossible. On a good day SD cards can write a few hundred kB per second (my experience...others on this forum are more expert and can probably quote more concrete numbers).


Right, yes, sorry; got my ns mixed with my us. I meant to write microseconds, but I guess that's still impossible anyway.


It definitely makes sense to write data in 512-byte chunks, and using interrupts to log the data to 1 buffer while writing a 2nd buffer to SD is a good approach. Setting aside two 512-byte buffers uses up half of the UNO's 2kB of RAM. So it's doable, but leaving some room for stack space means you only have about 500-700 bytes of RAM left to play with. That might be enough, depending on your application.


Great, thanks for that info. I'm worried, though, that interrupts would mess with the timing of the code; and logging the data with high resolution, accurate, time is a key specification. Actually, what about storing to the Flash memory? Would that slow anything down, do you think?
Title: Re: Fastest Data Logging? With SD Card?
Post by: graynomad on Mar 29, 2012, 01:03 pm
You can't store into flash so that's out.

The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).

_______
Rob
Title: Re: Fastest Data Logging? With SD Card?
Post by: Jantje on Mar 29, 2012, 01:49 pm
Maybe an idea
It may be an option not to write all data.
If the data is often the same as the previous data you may opt to write a repeat value instead of a new value.
So if your data is 1 1 2 2 2 2 2 3 3 3 3 4 4 4 4 you could write all data
1,1,2,2,2,2,2,3,3,3,3,4,4,4,4,5
or optimized data
1*2,2*5,3*4,4*4,5*1
Note that if you optimize this (never write *1) you will always save on writes.

However I am wondering that as you need this high frequency of reads whether repeats are expected. If not it is probably not worth the cpu cycles.
If you want this high frequency to notice spikes is it worth saving all the data?

As to the interrupt. Given the potential delays on SD cards a interrupt is your only guarantee for not missing reads.

Best regards
Jantje
Title: Re: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Mar 29, 2012, 06:37 pm

You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

Quote from: [url=http://arduino.cc/en/Reference/PROGMEM]http://arduino.cc/en/Reference/PROGMEM[/url]
http://arduino.cc/en/Reference/PROGMEM (http://arduino.cc/en/Reference/PROGMEM):
Store data in flash (program) memory instead of SRAM.



It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.


The dual buffering system mentioned seems the best idea but if the data is constant you won't have time to save it somewhere. And yes interrupts will introduce some jitter, better off without them.

Exactly what sample rate do you need and for how long? (Hint, you can have fast and you can have long but not both).

I have yet to receive the final design specs, but it needs to be at least  in the single digit milli second time.

I know that the Arduino is capable of microsecond time, and, in fact, according to the wisdom of the ancients, it is capable of at least 56KHz read speeds. http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1208715493/15

On top of that, there used to be a library that could move data at incredibly fast speeds as well; but it has since 404'd...
http://arduinonut.blogspot.com/2009/02/libraries.html

But, there's another way using the i2c, increasing the read/write speed to 400KHz: http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1241668644/0

Hmmm.
Title: Re: Fastest Data Logging? With SD Card?
Post by: wildbill on Mar 29, 2012, 07:02 pm
Quote
I don't think that's true. What about with the prgspace library?


That's for storing read only data that you set at compile time - you can't write on progmem at runtime.
Title: Re: Fastest Data Logging? With SD Card?
Post by: Jantje on Mar 29, 2012, 07:37 pm


You can't store into flash so that's out.

I don't think that's true. What about with the prgspace library?

In my humble opinion you should rethink



It may be an option not to write all data.

Unfortunately, not this time; I'm trying to do time measurements of microsecond pulses; essentially the times between pulses, so I can't omit much of the data.

In my humble opinion you should rethink
Title: Re: Fastest Data Logging? With SD Card?
Post by: graynomad on Mar 29, 2012, 11:17 pm
Quote
But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

Quote
at least  in the single digit milli second time.

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.

______
Rob
Title: Re: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Mar 30, 2012, 02:21 pm

That's for storing read only data that you set at compile time - you can't write on progmem at runtime.


Ahh... Must have misread that one.


Quote
But, there's another way using the i2c, increasing the read/write speed to 400KHz:

But where will you move it to?

To the SD card..? Or am I really misunderstanding everything, in blind hope?  :smiley-red:

Quote

1-9mS, that's heaps of time for the processor to read and store data in general, but I don't know much about SD card timing.

If you can save say 512 bytes to the SD in < 512 x sample-rate then the above-mentioned dual buffer idea should work. And despite my reluctance to use interrupts they may be required in this case.

You set a timer up to interrupt every (say) 5mS, the ISR does a reading and stores in buffer A. If the buffer is full it sets a flag.

The main code waits for the flag, when it is set it swaps a pointer to the buffer B and proceeds to save buffer A. 512 samples later the procedure is repeated with the pointer being swapped back to buffer A.


Ok, so this idea does sound the best so far, but I think I'm unclear on the benefit of an interrupt..

1. Set an interrupt that checks every 5ms
2. Gather data in buffer A until 512
3. Flag is set, and when interrupt check is called, it writes the data...?
4. Data in buffer B is gathered, repeat...?

What makes an interrupt check better than having a looping check?
If I'm checking to see if buffer A is full to switch to buffer B, why is an interrupt better than if I were to simply write buffer A to the card in that same if block?
Title: Re: Fastest Data Logging? With SD Card?
Post by: graynomad on Mar 30, 2012, 02:35 pm
Quote
Or am I really misunderstanding everything, in blind hope?

Yep, I2C can't talk to an SD card. Normally you use SPI.

Quote
I think I'm unclear on the benefit of an interrupt..

Quote
simply write buffer A to the card in that same if block?

You could check in a loop and that may work, I'm not sure how the SD library operates. But if you call it to write 512 bytes and it blocks until the job is done you would miss some data. OTOH if the write is fast enough and/or the library doesn't block then you are OK. Hopefully someone who knows more about the SD library can help with that.

If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

______
Rob
Title: Re: Fastest Data Logging? With SD Card?
Post by: Constantin on Mar 30, 2012, 03:54 pm
You may be using the wrong microcontroller for the task.

My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. at a similar hardware price and with pretty good IDEs as well. That should make sampling the amount of data you're envisioning a lot simpler than trying to make a Arduino do what it likely cannot (at least with the constraints you're envisioning).

But that's just me. FWIW, I recall reading that the smallest time slice a Arduino can offer is on the order of 65ns at 16MHz. That's for a single instruction. So, short, bursty signals may be better sampled on a higher-speed (i.e. SPI based) ADC that offers speed and resolution.
Title: Re: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Mar 30, 2012, 06:56 pm



You may be using the wrong microcontroller for the task.
My suggestion would be to graduate to a Maple, ChipKit, or similar board that offers oodles of RAM, high clock speeds, etc. 


Thanks for the suggestions; maybe going another route is a plausible solution, as long as the prices are similar.



Yep, I2C can't talk to an SD card. Normally you use SPI.
If you use interrupts it's like having two separate jobs running so as long as the SD code doesn't disable the interrupts you will have a small amount of jitter but overall it should run smoothly.

Bottom line, we need to know more about writing to SD cards on the Arduino.

I see. So interrupts can possibly work concurrently? That's pretty great, then. RuggedCircuits' original solution will be the way to go.
Also, I've heard there's a faster alternative to the standard arduino library that I can use; is this true?
Title: Re: Fastest Data Logging? With SD Card?
Post by: graynomad on Mar 31, 2012, 01:28 am
Quote
I've heard there's a faster alternative to the standard arduino library that I can use; is this true?

I think there is, but don't know where to get it.

This should be doable on the Ardiuno if the write speeds to the SD are fast enough.

RC said

Quote
On a good day SD cards can write a few hundred kB per second


You should be able to look at your requirements and do the maths based on that.

Let's be conservative and say 100kbps, at that speed you can write a 512-byte block in 5mS so we're in the ball park.

_______
Rob
Title: Re: Fastest Data Logging? With SD Card?
Post by: graynomad on Mar 31, 2012, 03:59 pm
Here's an SD library by fat16lib

http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html (http://arduino.cc/forum/index.php/topic,98639.0/topicseen.html)

He's quoting nearly 200 KB/sec

_____
Rob
Title: Re: Fastest Data Logging? With SD Card?
Post by: pito on Mar 31, 2012, 04:34 pm
..pls mind an sdcard can have ~250ms writing latency, so you need a buffer where you can store 250ms worth of data.. sdcard is not a linear memory you can push data at given rate into.. if you need a data logger for submicroseconds events you must know:
1. what is the byte size of a single sample (ie adc=14bit = 2bytes)
2. sample/event duration (ie a .2usec pulse)
3. sampling/event's frequency (ie 1MHz)
4. for how long you going to sample and store (ie 100days)
5. etc.
Then you may start to think on technology to be used. It is called a "detailed requirements specification".. :)
Title: Re: Fastest Data Logging? With SD Card?
Post by: Jantje on Mar 31, 2012, 05:15 pm
This means you should have 3 buffers just in case when you hit this delay when you need to swap buffers.
Best regards
Jantje
Title: Re: Fastest Data Logging? With SD Card?
Post by: pito on Mar 31, 2012, 05:36 pm
For example: pic24/dspic33 (40-70MHz mcus) - you can sample at 1Msamples/sec (10bit, but let say 8bits to keep it simple), ADC is connected via DMA to two 2kB RAM blocks (buf1 and buf2) in so called ping-pong mode (ie you fill in the buf1 from the ADC and at the same time you process the buf2, and vice versa, all is done in hw automaticaly) - so you have 2msecs to process one buffer and to feed its content into buffer3 (ie via second DMA channel), which needs to contain worst case up to 250ms of data - that is 250kB of data. This could be ie an external SRAM, FRAM or MRAM (256kx8, serial/parallel) device. Then you need to move that buffer3 (or its actual content according to the watermark, ie via third DMA channel) into the sdcard (here you need a buffer4 - 512bytes or more for one or more sdcard's blocks). This will work when writing speed to the sdcard via SPI will be higher than 1MByte/sec (to empty the buffer3 fast - here probably an SDIO interface would be a better choice).
So this is maybe a "feasible" situation. If you are going to store into an external RAM device, ie serial/parallel SRAM, you do not need the buffer3..
Not doable with arduino, however. p.
Title: Re: Fastest Data Logging? With SD Card?
Post by: fat16lib on Mar 31, 2012, 07:36 pm
Write latency is the big problem with SD cards.  SD cards are designed to do large contiguous writes.  Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated.  This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands.  This reduces the maximum write latency.  I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency.  8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at http://code.google.com/p/beta-lib/downloads/list (http://code.google.com/p/beta-lib/downloads/list).

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html (http://arduino.cc/forum/index.php/topic,64813.0.html)

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384 (http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384).



Title: Re: Fastest Data Logging? With SD Card?
Post by: pito on Mar 31, 2012, 08:43 pm
@fat16lib: what is the typical wr latency with that sdfat feature, pls?
Title: Re: Fastest Data Logging? With SD Card?
Post by: fat16lib on Mar 31, 2012, 09:20 pm
There is not really a typical write latency.  It depends on the card design, its history of use, how it was formatted, how fast you are trying to write in blocks/second.

I wrote a test program, SdFatRawWrite.  It is included with SdFat in the extra/examplesV1/SdFatRawWrite folder.   SdFat has had raw write for a long time.

I like SanDisk cards, 1 GB or 2 GB formatted FAT16. 

I ran SdFatRawWrite on an old 1 GB Sandisk Ultra II card formatted FAT16. I wrote a 10 MB file at 102,400 bytes per second.  The result was:

Quote

Start raw write of 10240000 bytes at
102400 bytes per second
Please wait 100 seconds
Done
Overruns: 0
Elapsed time: 100000 millis
Max write time: 828 micros


This card has extremely good performance.   The max time to write a 512 byte block was 828 usec.

This card was manufactured in 12/2006.  Many newer cards can't beat it.
Title: Re: Fastest Data Logging? With SD Card?
Post by: pito on Mar 31, 2012, 09:35 pm
I see.. We've spent a lot of time elaborating the sdcard stuff (see retrobsd.org ) as we use the sdcard for bsd unix filesystem and swap file as well. With pic32 we have achieved 1.7MBytes/sec read (SPI, no DMA) and ~800kB/sec write afaik, but with time (the swap swaps about 200kB/sec data based on number of jobs running) some of my cards started to become very slow after few weeks. The fastest cards are 256MB and 512MB ones, but hard to find today..
Title: Re: Fastest Data Logging? With SD Card?
Post by: fat16lib on Mar 31, 2012, 10:03 pm
Once again, write speed is the wrong measure for data logging with limited RAM.  BSD swap and file system performance is irrelevant.

The limit is the amount of buffer memory and the maximum write latency for a block.  Buffer size divided by max latency is the speed limit.

The write speed spec for high speed SDHC cards is based on writing many MB of data.  This is reasonable for video cameras that have huge buffers.  These cards may have poor performance in Arduino data logging since so little RAM is available to handle the occasional long write latency these cards have.

Edit:

The above card has a very low write latency as long as you allow at least four milliseconds between writes.  If you push it to one block every three milliseconds it will have an occasional 14-15 millisecond latency.  So the max data logging rate has little to do with the max write rate.   I assume you don't want to miss data points.

If you write this card at max speed, the rate is 428 KB/sec but there are now occasional 44 millisecond write latencies.
Title: Re: Fastest Data Logging? With SD Card?
Post by: Neodudeman on Apr 01, 2012, 01:41 am

Write latency is the big problem with SD cards.  SD cards are designed to do large contiguous writes.  Libraries like SdFat write files using single block write commands and do not do contiguous writes since file structures like the FAT must be updated.  This causes an occasional large write latency.

I have added features to SdFat to allocate large contiguous files and write these files with efficient contiguous multi-block write commands.  This reduces the maximum write latency.  I have logged 12-bit MCP3201 ADC data at 40,000 samples per second which requires 80 KB/sec writes with very low latency.  8-bit MCP3001 ADC data can be logged at up to 90,000 samples per second.

The technique is demonstrated in the binaryLogger sketch. binaryLogger is in fastLoggerBeta20110802.zip at http://code.google.com/p/beta-lib/downloads/list (http://code.google.com/p/beta-lib/downloads/list).

Also see this topic http://arduino.cc/forum/index.php/topic,64813.0.html (http://arduino.cc/forum/index.php/topic,64813.0.html)

Here is a plot of 40,000 sample per second data http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384 (http://forums.adafruit.com/viewtopic.php?f=31&t=21629&p=113384#p113384).


Thank you, everyone! Your help is greatly appreciated. I've taken all your advice, and will ponder on it for a while.

And, a special thanks to you, Fat16Lib.
Because of hard working, generous gurus like yourself, arduino-laymen like myself can utilize the wisdom of the masters!
Title: Re: Fastest Data Logging? With SD Card?
Post by: adamatcooranbong on Apr 27, 2013, 02:37 am
Hi Guys,

Only new to the Arduino world so please take easy on me...

I'm having issues writing data to a 1 GB SD card. I have formatted it to FAT 16. I'm using a Mega ADK with the WiFi shield to log data to the SD.

It a very basic setup... trim-pot resistor connected to A0. All I want to do is read and log the voltage to a .csv file - voltage needs to be a float but have tried it as an integer with no increase in logging speed.

The speed of logging the data to the SD is very very slow - about 4 s per sample. For what I have read the limitation of the SD card is in the order of ms not s. I was thinking is has something to do with the buffer becoming full but for what I have read the .close() function should clear the buffer.

Can somebody please tell me why my code is so slow or load it for themselves and see if they experience the same issues?

Any help would be much appreciated.

Cheers

Adam
Title: Re: Fastest Data Logging? With SD Card?
Post by: Nick_Pyner on Apr 27, 2013, 04:00 am
I have looked at your code and can't see what you are doing with strings, hopefully nothing.

You may find the SD is fine but your hardware is struggling with the serial rate. Since you have a 1 sec delay in the loop there can't be a very good reason for not using the default 9600.

I use pin 10, output for both Uno and plain vanilla Mega but I guess if this was a problem it wouldn't work at all and what you have is kosher for the ADK Mega.

I don't understand the
Code: [Select]
  }
  else if(id>10)
  {
    return;
  }
  else
  {
    Serial.println("Could not open file...");
    return;
  }


But, if you eventually get what you want, I guess it's kosher.

Title: Re: Fastest Data Logging? With SD Card?
Post by: adamatcooranbong on Apr 27, 2013, 10:28 am
Hi Nick,

Thanks for your comments.

Regarding the bit of code that didn't makes sense to you... just some error checking while the program is still in the development phase. I wouldn't have the program stop until there was an input from the user for the real version.

In the current form the program only collects 10 samples. I can manage to collect and log up to 32 samples quickly i.e. order of ms to csv however; anymore samples seems to make the program hit a brick wall. Sample rate decreases dramatically and csv file becomes corrupted.

The else statement just see if the file can be opened and if it can't displays the error to the user.

I have tested the program with and without the delay(1000) and it doesn't seem to have any effect of the sample rate after > 30 samples.

I have also changed the baud rate back to 9600 and still not luck getting the thing to work effectively.

What's a "kosher"?

As far as I know there is little difference between the normal and ADK Mega boards in regards to their pin mapping. The following link states that pin 53 (SPI SS pin) on the Mega must be set as an OUTPUT - not pin 10. So I don't see how your code can work on the Mega.

http://arduino.cc/en/Main/ArduinoBoardMega2560

I only use the strings for serials comms and to write headers at the tops of the columns in the csv file. No string are used during the data collection and logging to csv.

Any other ideas mate?
Title: Re: Fastest Data Logging? With SD Card?
Post by: Nick_Pyner on Apr 27, 2013, 06:09 pm
I'm starting to run short of them quite quickly. I believe I have been  living in something of a fools' paradise and your post is confirming it. Last year I was obliged to move from Uno to Mega and sorted out the clock and analogue pin calls. You are right in that 53 is the SPI SS pin, and 10>13 do not carry the SPI.  Nonetheless, by carrying the code over from the Uno, I have pinmode 10,output and it has been working just fine for months.

I'm not sure I have an explanation for this. It could be because I meticulously keep the SD operations tightly bound together, to the exclusion of all else.

SD.open tralala
dataFile.print......
dataFile.print......
dataFile.close

I don't recall a technical reason for doing this, or even when I started doing it,  it just looks good on paper.  Having the same tight operation can't possibly be a bad idea, but I wouldn't recommend calling pin 10 to anybody.

I deleted your two string commands and the file compiles OK, which is just as well. It seems that anything to do with data strings is either poisonous, a waste of time, or both. More to the point, I now understand that your programme works fine initially and then hits a brick wall after 32 samples.  This might indicate that you are running out of memory. I have never experienced this, principally because I never accumulate data, it just passes through on its way to SD, LCD, Bluetooth, and the internet.  Problem is, your code isn't very different. The only thing I can see that I clearly wouldn't do  is the line

long id=1

indeed I would be very suss about it.  The way I see it, it means you could count seconds up to 2,147,483,648 i.e. for 68 years, before the variable is exhausted. There are likely to be memory implications here and you might try using a simple

int id=1
Title: Re: Fastest Data Logging? With SD Card?
Post by: adamatcooranbong on Apr 28, 2013, 07:26 am
Hi Nick,

I have managed to get the code going great now. There was an error in the code for using the WiFi sheild as a SD card reader/logger only.

I did a little more reading and found out that not only do you need pin 53 set as an OUTPUT but if you want to use the SD function of the shield explicitly used must deselect the WiFi chip (HDG104). This is done by setting pin 10 as an HIGH OUPUT. Apparently if I imported the WiFI library I would need to explicitly deselect the chip.

Code: [Select]
// libraries used
#include <SD.h>

// declare global constants and their data types
const int trimpotSensor=A0; // variable resistor on analog pin 0
const int chipSelect=4; // Chip Select pin default channel on EtherMega board for SD communication

// declare global variables and their data types
int sensorValue;
int id=1;
float volts;

// declare files
File dataFile;

// setup code here, to run once
void setup()
{
  Serial.begin(9600); // initialize serial communication
  while(!Serial)
  {
  }
  Serial.print("Initializing SD card..."); // print to the serial monitor a message
  pinMode(53,OUTPUT); // pin 53 must be left as an OUTPUT for the SD library to work
  pinMode(10,OUTPUT); // pin 10 must be left as an OUTPUT so it can be set HIGH
  digitalWrite(10,HIGH); // pin 10 needs to be put HIGH to explicitly deselect the WiFi HDG104 chip
  if(!SD.begin(chipSelect)) // if the statement !SD.begin(SS) comes back as FALSE execute the "if" statement
  {
    Serial.println("Card failed or not present"); // print to screen the SD card status
    return; // start the loop again and check if the SD is faulty or not installed
  }
  else
  {
    Serial.println("Card initialized"); // else everything is all good
  }
  if(SD.exists("volts.csv"))
  {
    Serial.println("File exists and now being removed");
    SD.remove("volts.csv");
  }
  else
  {
    Serial.println("No file could be found");
  }
  Serial.print("Sample No.");
  Serial.print(",");
  Serial.println("Volts");
  dataFile=SD.open("volts.csv",FILE_WRITE);
  dataFile.print("Sample No.");
  dataFile.print(",");
  dataFile.println("Volts");
  dataFile.close();
}

// main code
void loop()
{
  sensorValue=analogRead(trimpotSensor);
  volts=5.0/1024.0*float(sensorValue);
  dataFile=SD.open("volts.csv",FILE_WRITE);
  if(dataFile && id<=50)
  {
    delay(150);
    dataFile.print(id,DEC);
    dataFile.print(",");
    dataFile.println(volts,DEC);
    dataFile.close();
    Serial.print(id,DEC);
    Serial.print(",");
    Serial.println(volts,DEC);
  }
  else
  {
    return;
  }
  id++;
}


Also found out the SD card I was using was pretty crappy. Got myself a good 8 GB ScanDisk SD and it can log data in the order of ms now. So all is good for now.

Thanks a lot for your assistance. Also changed the counter to an int variable so that could also be a factor.

Cheers

Adam
Title: Re: Fastest Data Logging? With SD Card?
Post by: drkblog on Oct 03, 2013, 02:18 am
Hello guys! I know this post is somewhat old but is very much related to my question:

I have to write to an SD card 307,200 bytes at 1 byte per microsecond (max). I'm testing the RawWrite example from SdFat library setting 512 µS per block (512 bytes) but the best I see in the log is 708 µS per block. In fact, the total time is more than twice the expected time, so I guess the average write is above 1024 µS. The max write time is 35 mS which is far more than my application limit.

I have to dump 307,200 bytes from a AL422B FIFO to the SD. This chip supports a minimum reading clock of 1MHz.

Based in the numbers I've seen in this post and the few tests I've made, there is no way I can do this. But just in case, is there anyone there who got better results?
Title: Re: Fastest Data Logging? With SD Card?
Post by: Jantje on Oct 03, 2013, 11:52 am
Which sd card are you using.
Title: Re: Fastest Data Logging? With SD Card?
Post by: koksy123 on Nov 14, 2014, 12:50 am
Hi Guys,

I have a very similar problem where I have to store sampled data of a low frequency SineWave of 250Hz into a SDcard via the Arduino WiFi Shield. Note that I am relatively new to the Arduino world.

Detailed Requirement Specs:
1) Sampling Frequency = ~8kHz (rate of which ADC samples data)
2) Byte Size of single sample = 10 bits adc = 1.25bytes
3) SDcard = Micro SanDisk card 32GB.

I did do a lot of research on this and I know that SDcard writing latency is about ~250ms for SDHC. I don't think I will be able to comprehend advance technique such as changing the kernel. All i need to do is to sample this 250Hz or less analog wave and to be stored in the SD card. Also, can the Arduino ADC's sampling rate be reduced below 8kHz?

Any help would be awesome. I really need this project done ASAP.

THANK YOU.