Slow Writing to SD

Hi All,

I'm using an Arduino Mega 2560 R3 and an Adafruit Data Logging Shield to record data from 2 LIS3DH accelerometers. Currently, the fastest updates I can get is around 120Hz. I need to be able to record around 800 - 1200Hz. The sensors should support that. I'm thinking there is something in my sketch that is preventing faster reading/writing. Writing to the SD card could also be another problem, but I'm recording several records before writing so if that were the only problem I'd expect to see timestamps at faster intervals with periodic long intervals when writing. I've included the sketch and a fritzing for your review. Also, the SD card is a 16gb ADATA 4c.

// based on https://learn.adafruit.com/adafruit-data-logger-shield/using-the-real-time-clock-3

#include <Adafruit_LIS3DH.h>
#include <Adafruit_Sensor.h>
#include <SPI.h>
#include <SD.h>
#include <Wire.h>
#include <RTClib.h>

// A simple data logger for the Arduino LIS3DH

// how many milliseconds between grabbing data and logging it. 1000 ms is once a second
#define LOG_INTERVAL  2.5 // mills between entries (reduce to take more/faster data)

// how many milliseconds before writing the logged data permanently to disk
#define SYNC_INTERVAL 1000 // mills between calls to flush() - to write data to the card
uint32_t syncTime = 0; // time of last sync()

#define WAIT_TO_START    0 // Wait for serial input in setup()

// the digital pins that connect to the LEDs
#define redLEDpin 2
#define greenLEDpin 3

// Used for software SPI
#define LIS3DH_CLK 13
#define LIS3DH_MISO 12
#define LIS3DH_MOSI 11

// Used for hardware & software SPI
#define LIS3DH1_CS 10
#define LIS3DH2_CS 8

Adafruit_LIS3DH accel_1 = Adafruit_LIS3DH(LIS3DH1_CS, LIS3DH_MOSI, LIS3DH_MISO, LIS3DH_CLK);
Adafruit_LIS3DH accel_2 = Adafruit_LIS3DH(LIS3DH2_CS, LIS3DH_MOSI, LIS3DH_MISO, LIS3DH_CLK);

const int s1 = 10;
const int s2 = 8;

RTC_PCF8523 RTC; // define the Real Time Clock object

// for the data logging shield, we use digital pin 10 for the SD cs line
const int chipSelect = 53;

// the logging file
File logfile;

void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);

  // red LED indicates error
  digitalWrite(redLEDpin, HIGH);

  while (1);
}

void setup(void)
{
  Serial.begin(9600);
  Serial.println("Accelerometer Write Test"); Serial.println("");

  digitalWrite(s1, HIGH);
  digitalWrite(s2, HIGH);

  /* Initialise the sensor */
  if (!accel_1.begin() | !accel_2.begin() )
  {
    /* There was a problem detecting LIS3DH ... check your connections */
    Serial.println("Ooops, LIS3DH not detected ... Check your wiring!");
    while (1);
  }

  /* Set the range to whatever is appropriate for your project */

  char g_range = LIS3DH_RANGE_16_G;
  accel_1.setRange(g_range);
  accel_2.setRange(g_range);

  char freq = LIS3DH_DATARATE_LOWPOWER_5KHZ; //Changing this doesn't seem to have any effect
  accel_1.setDataRate(freq);
  accel_2.setDataRate(freq);

  // use debugging LEDs
  pinMode(redLEDpin, OUTPUT);
  pinMode(greenLEDpin, OUTPUT);

#if WAIT_TO_START
  Serial.println("Type any character to start");
  while (!Serial.available());
#endif //WAIT_TO_START 

  // initialize the SD card
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(53, OUTPUT);

  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    error("Card failed, or not present");
  }
  Serial.println("card initialized.");


  // create a new file
  char filename[] = "LOGGER00.CSV";
  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = i / 10 + '0';
    filename[7] = i % 10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE);
      break;  // leave the loop!
    }
  }

  if (! logfile) {
    error("couldnt create file");
  }

  Serial.print("Logging to: ");
  Serial.println(filename);

  // connect to RTC
  Wire.begin();
  if (!RTC.begin()) {
    logfile.println("RTC failed");
#if ECHO_TO_SERIAL
    Serial.println("RTC failed");
#endif  //ECHO_TO_SERIAL
  }


  logfile.println("millis,accel_1x, accel_1y, accel_1z,accel_2x,accel_2y,accel_2z");
}

void loop(void)
{
  DateTime now;

  // delay for the amount of time we want between readings
  delay(LOG_INTERVAL);

  digitalWrite(greenLEDpin, HIGH);

  // log milliseconds since starting
  uint32_t m = millis();
  logfile.print(m);           // milliseconds since start
  logfile.print(", ");
  accel_1.read();      // get X Y and Z data at once
  accel_2.read();

  logfile.print(accel_1.x);
  logfile.print(", ");
  logfile.print(accel_1.y);
  logfile.print(", ");
  logfile.print(accel_1.z);
  logfile.print(", ");
  logfile.print(accel_2.x);
  logfile.print(", ");
  logfile.print(accel_2.y);
  logfile.print(", ");
  logfile.print(accel_2.z);
  logfile.print("\n");

  digitalWrite(greenLEDpin, LOW);

  // Now we write data to disk! Don't sync too often - requires 2048 bytes of I/O to SD card
  // which uses a bunch of power and takes time
  if ((millis() - syncTime) < SYNC_INTERVAL) return;
  syncTime = millis();

  // blink LED to show we are syncing data to the card & updating FAT!
  digitalWrite(redLEDpin, HIGH);
  logfile.flush();
  digitalWrite(redLEDpin, LOW);
}

I'm willing to consider all alternatives including alternative platforms like Raspberry Pi etc. Thank you for any help!

(deleted)

Thank you for taking a look. I've checked that out and it didn't make any real difference, maybe an additional 40Hz.

(deleted)

You might take a look at the LowLatencyLogger.ino example in the SdFat library. It seems to do everything possible to speed up the process. It creates a new file that's 125MB in size, then erases the data portion of the file (so subsequent writes don't require an erasure first), then simply writes directly to sequential sectors to log the data, bypassing the file system entirely. It's only when you stop that it truncates the file to what you've actually saved, and adjusts the directory and FAT entries accordingly. It uses multiple ram buffers so you can continue to fill a new buffer while writing the old one(s) to the card. The example logs at 500Hz, but it may well do much more on a Mega. You might try this out on your Mega to see how fast it will actually go with your card. I think that would represent the limit of what is possible. The comments at the beginning say 4000 Hz should be possible with a Due, but of course that would also depend on the card.

Thanks, running a quick test without my accelerometers I can get it to write fast enough. Now I have to figure out how to convert it to work with my accelerometers! Oh boy, this might be rough...