SDfat slow buffer flush

I am using the SDfat library to log 4 analog inputs and an IMU (MPU6050).

I am looking to get a stable logging frequency of around 100 Hz. Currently I am sending 40 data samples to the buffer before flushing to the SD card and without the delay of 5 milli seconds I am able to log up to around 800 Hz.

The problem I am having is the time taken to flush values to the SD. It takes approx. 14 ms to flush the values meaning I miss capturing data during that period. When sending and flushing 1 value per loop the maximum frequency I can get is only around 60Hz.

I wonder if anyone knows of a more efficient way of flushing the values to the card?

Please see code below.

Thanks.

//--------------Include Libraries----------------------
#include <SdFat.h>    // SD card library
#define CHIP_SELECT SS  // SD chip select pin
#include "I2Cdev.h"
#include "MPU6050.h"

#if I2CDEV_IMPLEMENTATION == I2CDEV_ARDUINO_WIRE
#include "Wire.h"
#endif

//----------------------------------------------------

//-----------------Declerations-----------------------
// class default I2C address is 0x68
// specific I2C addresses may be passed as a parameter here
// AD0 low = 0x68 (default for InvenSense evaluation board)
// AD0 high = 0x69
MPU6050 accelgyro;
//MPU6050 accelgyro(0x69); // <-- use for AD0 high

int16_t ax, ay, az;
int16_t gx, gy, gz;

// uncomment "OUTPUT_READABLE_ACCELGYRO" if you want to see a tab-separated
// list of the accel X/Y/Z and then gyro X/Y/Z values in decimal. Easy to read,
// not so easy to parse, and slow(er) over UART.
#define OUTPUT_READABLE_ACCELGYRO

// uncomment "OUTPUT_BINARY_ACCELGYRO" to send all 6 axes of data as 16-bit
// binary, one right after the other. This is very fast (as fast as possible
// without compression or data loss), and easy to parse, but impossible to read
// for a human.
//#define OUTPUT_BINARY_ACCELGYRO

// Create file system object
SdFat sd;

// Define text file for logging
ofstream logfile;

// Create a buffer variable
char buffer[6700];

//----------------------------------------------------------------------------------

void setup() {
  // join I2C bus (I2Cdev library doesn't do this automatically)
#if I2CDEV_IMPLEMENTATION == I2CDEV_ARDUINO_WIRE
        Wire.begin();
#elif I2CDEV_IMPLEMENTATION == I2CDEV_BUILTIN_FASTWIRE
        Fastwire::setup(400, true);
#endif
      accelgyro.initialize();

// initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!sd.begin(CHIP_SELECT, SPI_FULL_SPEED )) sd.initErrorHalt();
  
  // Create a new file in the current working directory
  // and generate a new name if the file already exists
  char fileName[13] = "LOGGER00.CSV";
  
  for (uint8_t i = 0; i < 150; i++) 
  {
    fileName[6] = i/10 + '0';
    fileName[7] = i%10 + '0';
    if (sd.exists(fileName)) continue;
    logfile.open(fileName);
    break;
  }
  // Connect the buffer to the variable buffer
  obufstream bout(buffer, sizeof(buffer)); 
  
  // Write a line for the headers
  bout << pstr("time_milli_gps");
  bout << pstr(",Lat");
  bout << pstr(",Long");
  bout << pstr(",Date");
  bout << pstr(",GPS_Time");
  bout << pstr(",Altitude_m");
  bout << pstr(",Course_Deg");
  bout << pstr(",Speed");
  bout << pstr(",time");
  bout << pstr(",accel_x");
  bout << pstr(",accel_y");
  bout << pstr(",accel_z");
  bout << pstr(",Gyro_x");
  bout << pstr(",Gyro_y");
  bout << pstr(",Gyro_z");
  bout << pstr(",pot_1");
  bout << pstr(",pot_2");
  bout << pstr(",pot_3");
  bout << pstr(",pot_4");
  bout << pstr(",Emergency_Stop");
  bout << pstr(",Button_of_Choice");
  bout << endl;
  
  logfile << buffer << flush;
}



void loop() {
  // read raw accel/gyro measurements from device
    accelgyro.getMotion6(&ax, &ay, &az, &gx, &gy, &gz);
  
  // Use buffer stream to format line
    obufstream bout(buffer, sizeof(buffer));
    
  // Create a loop to read a block of data and send them to the RAM 
   for (uint8_t ii = 0; ii < 40; ii++)
   { 
      // Acquire data and copy on buffer
      bout << pstr("time_milli_gps");
      bout << pstr(",Lat");
      bout << pstr(",Long");
      bout << pstr(",Date");
      bout << pstr(",GPS_Time");
      bout << pstr(",Altitude_m");
      bout << pstr(",Course_Deg");
      bout << pstr(",Speed");
      bout << ',' << micros();
      bout << ',' << ax; // FLz
      bout << ',' << ay; // FRz
      bout << ',' << az; // RLz
      bout << ',' << gx; // RRz
      bout << ',' << gy; // RRz
      bout << ',' << gz; // RRz
      bout << ',' << analogRead(A0);
      bout << ',' << analogRead(A1);;
      bout << ',' << analogRead(A2);
      bout << ',' << analogRead(A3);
      bout << pstr(",E_stop_s");
      bout << pstr(",button_s");
      bout << endl;
      delay(5); // 5 msec = 200 Hz
   } 
   
   logfile << buffer << flush;
}

I miss capturing data during that period.

The problem is that write latency for SD cards can occasionally be very long. This is caused by various housekeeping functions in the SD card.

SD cards must erase flash before writing. SD cards perform wear leveling to increase the life of the SD card.

You must separate data capture from writing to the SD card and buffer data waiting to be written.

You can capture data in an interrupt routine like this example Try this super fast analog pin logger - Storage - Arduino Forum .

You can use a small RTOS like the nilSdLogger example in NilRTOS Google Code Archive - Long-term storage for Google Code Project Hosting. NilRTOS20130720.zip.

Your current program will have so much time jitter that there is no point in logging data this fast. The information in a time series depends as much on the accuracy of the time of a data point as the accuracy of the value.

Many articles have been published about time jitter noise. Google for "time jitter noise adc" for articles relevant to logging analog data.

If your data set is small, you could use a RAM disk RamDisk - a file system library for RAM devices like 23LCV1024 - Storage - Arduino Forum or RamDisk - Fujitsu 256 KB FRAM in 8-pin DIP - Storage - Arduino Forum. You will still need to solve the time jitter problem.

I am looking to get a stable logging frequency of around 100 Hz. Currently I am sending 40 data samples to the buffer before flushing to the SD card and without the delay of 5 milli seconds I am able to log up to around 800 Hz.
The problem I am having is the time taken to flush values to the SD. It takes approx. 14 ms to flush the values meaning I miss capturing data during that period. When sending and flushing 1 value per loop the maximum frequency I can get is only around 60Hz.

What is the data rate you want to log??

I would estimate the 4 analog and the 6 IMU might be:

(4 + 6) * 2 bytes = 20bytes
plus a timestamp ie. 4 bytes
total = 24bytes.

100x per second to log means 2.4kBytes/second data rate (my assumption here is the IMU is able to provide the data 100x per second).

An sdcard has the write latencies of dozens of milliseconds up to 250ms each XYZ milliseconds (almost random).

So in order to save 2.4kBytes/second to the sdcard, you need a FIFO buffer. The size of the FIFO buffer, worst case is:

FIFO_size = data_rate * worst_case_latency = 2.4kB/sec * 250msec = 600Bytes.

With a 600 Bytes large FIFO and the NilRtos you may (most probably) save your data on any sdcard.

PS: this topic has been discussed here many times. Search this forum for "Nilrtos Fifo"..

Thank you for your reply.

That's given me more than enough to progress with.