How to write data in chunks to SD card to improve Sampling Rate? [SOLVED!]

Hi

I'm working on a project that is sampling the output of a Strain Gauge attached to a Karabiner in order to measure the impulse forces of a fall and the time it occurs (relative to the arduino being switched on).

There may be multiple falls and for ease of access to the data I am using the Arduino Ethernet w/o POE as it has a SD card slot built on.

The simple program I have below does this for me:

/*
Final Code
 	 
 */

#include <SD.h>

// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.
const int chipSelect = 4;
int sensor;
unsigned long time;
void setup()
{
 // Open serial communications and wait for port to open:
  Serial.begin(9600);
   while (!Serial)
{
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");
}

void loop()
{
   time = millis();

  String dataString = "";

     sensor = analogRead(A0);
    dataString += String(sensor);
  
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  File dataFile = SD.open("datalog.txt", FILE_WRITE);

  // if the file is available, write to it:
  if (dataFile ) {

   // Value "100" refers to a change in voltage indicating a strain on the karabiner i.e a fall
    if (sensor > 100){  
   
    // Save the value and the time to SD card
    dataFile.println(dataString +"," +time);
    }
     dataFile.close();
    // print to the serial port too:
    Serial.println(dataString+"," +time);
  }  
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
  }
  
}

The issue I'm having is the sampling rate at the moment is around 40 values a second. In order to receive accurate results I would like to improve this to around 100 samples/second.

Looking through similar issues on the forum I believe the time it takes to write to the SD card is what is causing the slow sampling rate and in order to overcome this a buffer could be used, possibly the EEPROM on the Arduino and then write these values to the SD card in chunks.

Another issue other users had was during the time it took to write from the EEPROM to the SD card, chunks of data are lost. Fortunately I only need to save selected data and this is usually 1 seconds worth of data.

I was hoping someone could help me with the code needed to do this. I am an ME student and my programming skills are lacking to say the least.

Cheers for any advice folks.

A quick look at your code and possibly a bit of re-structuring will speed it up.
Your currently continually opening, writing and closing the SD file in the loop where a better method might be to wait until sensor reading is >100, note the time and sample x number of samples at y interval into RAM. You only need to log the sensor value (int) as your reading at a fixed time interval. A hundred reading will consume 200 bytes of RAM. After the readings have happened then open, write the values & close the SD card and print to Serial if needed.
Getting rid of the String class will probably help speed/stability issues as it is known to be buggy.

Thanks for your speedy reply. I will definitely try to fix the robustness of my program.

Riva:
After the readings have happened then open, write the values & close the SD card and print to Serial if needed.

If possible could you or anyone show me a simple example code of this, I'm having a hard time getting my head round being able to implement this

Cheers

I'm currently without hardware to test on but something like this...

#include <SD.h>

// On the Ethernet Shield, CS is pin 4. Note that even if it's not
// used as the CS pin, the hardware CS pin (10 on most Arduino boards,
// 53 on the Mega) must be left as an output or the SD library
// functions will not work.
const int chipSelect = 4;

const int ledPin = 13;

const int readingCount = 100; // Number of readings to take
const int readingDelay = 10;  // Sample interval in milliseconds
int readings[readingCount];   // Place to store readings

void setup() {
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }
  
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");

  pinMode(ledPin, OUTPUT);
  digitalWrite(ledPin,LOW); // Turn LED off
}

void loop()
{
  // Value "100" refers to a change in voltage indicating a strain on the karabiner i.e a fall
  Serial.println("Waiting...");
  
  while (analogRead(A0) < 100) {  
    // Wait till reading exceeds threshold
  }

  digitalWrite(ledPin,HIGH); // Turn LED on
  
  unsigned long time = millis();  // Note the start time
  for (int i = 0; i < readingCount; i++) {  // Take readingCount readings
    readings[i]=analogRead(A0);
    delay(readingDelay);
  }

  digitalWrite(ledPin,LOW); // Turn LED off
  
  time = millis() - time;   // Subtract current time to get sample duration
  // Calculate sample duration (Could be lazy and use readingDelay but this should be slightly more accurate)
  time = time / readingCount;   
  
  Serial.println("Saving...");
  // open the file. note that only one file can be open at a time,
  // so you have to close this one before opening another.
  File dataFile = SD.open("datalog.txt", FILE_WRITE);
  if (dataFile ) {
    // if the file is available, write to it:
    for (int i = 0; i < readingCount; i++) {  // readingCount readings
      // Save the value and the time to SD card
      dataFile.print(time * i);
      dataFile.print(",");
      dataFile.println(readings[i]);
      
      // print to the serial port too:
      Serial.print(time * i);
      Serial.print(",");
      Serial.println(readings[i]);
    }      
    dataFile.close();
  }
  // if the file isn't open, pop up an error:
  else {
    Serial.println("error opening datalog.txt");
    while (1);  // Stop
  }
}

All SD cards have occasional long write latencies. Special buffering techniques are required to log data at high rates.

See the example in this thread ¿como Unir nodo en eagle por nombre o valor? [SOLUCIONADO] - #4 by fmalpartida - Software - Arduino Forum.

Ah, I get it. Cheers Riva, I'll try making these changes to my code and I'll let you know what speed it get's bumped up to.

Fat16Lib, Thank you for your suggestion, I think what you said is beyond my programming skills and by the time I get it my project deadline will be up. I'll give it a bash in the next week or so and hopefully make some progress.

The Nil RTOS example is a working app. You don't need to program it for your use.

Just select the number of analog pins and rate. For one or two analog pins you can record at up to 1000 Hz.

The jitter in time between samples will be much better than a loop with a delay call. This could be important depending on what data analysis is required.