SDFat/SD: How long should SD.flush() take?

Hi everyone,

I'm writing program to log data from adafruit 9dof (a gyro, accelerometer and magnetometer breakout board) to an SD card. My intended rate is as close as possible to the 800Hz cited in the sensors description. I communicate with the sensor via I2C and with SD card via SPI, as shown here. I'm using an Arduino Nano.

It seems that the main bottleneck in the code is the .flush() command that pushes my buffer onto the SD card. At the time I am using a 256B buffer (I don't have enough SRAM for full 512B) allowing me to log 11 data points before having to flush the buffer. Each reading takes 1.4ms (or at 714Hz), but the flush requires almost 10ms, so my average sampling falls below 400Hz. This is already after accelerating I2C to 400KHz and having SPI as full speed. My data is stored as a binary struct to bypass any formatting overhead (as described here)

I am using the SDFat library, which performed around 4ms faster (I think) than SD, but this is still too slow, mainly because 10ms is a huge gap in readings.

My question is: is there anything I am missing that could accelerate the flush? Maybe I'm doing sth wrong?

I was thinking about implementing protothreads to be able to distribute my readings while logging data in the background, but I've never used them, so I'm not even sure whether it would be possible. Thoughts on that would be welcome. While not ideal, having <3ms spacing between samples would be satisfactory.

Finally, my code. Note that I also edited HardwareSerial.h in arduino\avr\cores and twi.h in arduino\avr\libraries\Wire\src\utility to change buffer size and I2C clock speed.

#include <Wire.h>
#include <SPI.h>
#include <SdFat.h>
SdFat SD;
#include <Adafruit_Sensor.h>
#include <Adafruit_FXAS21002C.h>
#include <Adafruit_FXOS8700.h>

// Sensor assignments
Adafruit_FXOS8700 accelmag = Adafruit_FXOS8700(0x8700A, 0x8700B);
Adafruit_FXAS21002C gyro = Adafruit_FXAS21002C(0x0021002C);


// Structs

// IMU measurement struct, note that doubles are 16-bit
struct IMUmeas {
    int acc [3];   // Accelerometer
    int gyro [3];  // Gyroscope
    int mag [3];   // Magnetometer
};

// Define the structure of the data packet
struct packet {
    struct IMUmeas IMU1;
    bool FSR[ 2 ];      // Force sensor
    uint32_t TS;  // Time Stamp in 10us
};

// Global vars
File dataFile;
struct packet datapoint;
long int initTime;
int count;
long int timer;

// Const Global vars;
const int TestPin = 4;
const int TestPin2 = 3;

// Function declarations
int CountFiles(void);

void setup() {

  // Define pins
  pinMode(TestPin, OUTPUT);
  pinMode(TestPin2, OUTPUT);
  
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
  }

  //Initialise SD card
  Serial.println("Initializing SD card...");

  if (!SD.begin(10)) {
    Serial.println("initialization failed!");
    while (1);
  }
  Serial.println("initialization done.");

  SD.remove("data.bin");

  // Open file on SD card
  dataFile = SD.open("data.bin", O_WRITE | O_CREAT);
  

//   Initialise gyro and print details
  Serial.println("Gyroscope Test"); Serial.println("");
  if(!gyro.begin())
  { Serial.println("Ooops, no FXAS21002C detected ... Check your wiring!"); }

  // Initialise accelerometer-magnetometer and print details
  Serial.println("Acceleromater and Magnetometer Test"); Serial.println("");
  if(!accelmag.begin())
  { Serial.println("Ooops, no FXOS8700 detected ... Check your wiring!"); }

  // Record initial time aka >>Start Timer<<
  initTime = micros();
  iter = 0;
  

}

void loop() {
  digitalWrite(TestPin, LOW);

  iter = iter + 1;

  for (count=0; count<11; count++)
  {
    digitalWrite(TestPin2, LOW);  
    
    // Retrieve data from sensor
    sensors_event_t gyrEvent;
    sensors_event_t accEvent;
    sensors_event_t magEvent;
    
    gyro.getEvent(&gyrEvent);
    accelmag.getEvent(&accEvent, &magEvent);
    
  
//  Save data to struct
    datapoint.IMU1.acc[0] = accEvent.acceleration.x; 
    datapoint.IMU1.acc[1] = accEvent.acceleration.y;  
    datapoint.IMU1.acc[2] = accEvent.acceleration.z;
    
    datapoint.IMU1.gyro[0] = gyrEvent.gyro.x; 
    datapoint.IMU1.gyro[1] = gyrEvent.gyro.y;  
    datapoint.IMU1.gyro[2] = gyrEvent.gyro.z;
    
    datapoint.IMU1.mag[0] = magEvent.magnetic.x; 
    datapoint.IMU1.mag[1] = magEvent.magnetic.y;  
    datapoint.IMU1.mag[2] = magEvent.magnetic.z;
    
    datapoint.FSR[0] = 0;
    datapoint.FSR[1] = 1;
    timer = (micros() - initTime)/10;
    datapoint.TS = (uint32_t) micros();
    
    dataFile.write((const uint8_t *)&datapoint, sizeof(datapoint));
    
    digitalWrite(TestPin2, HIGH);  
  }
  
  digitalWrite(TestPin, HIGH);  
  dataFile.flush(); 
}

Thank you very much for any input!

EDIT: I just realised there is a separate forum for storage, which would be more appropriate for this thread. I can't move or delete it myself, so I would like to kindly ask a moderator to move it. Sorry for the mistake.

Every other write of your 256 byte buffer automatically flushes the 512 byte segment to the card. So, why do it again?

Paul

I was convinced that datafile.write() will only move my data to the output buffer on the controller and l need datafile.flush() to actually push it to the card, no?

That was consistent with my observations: running code without the .flush() command wouldn't result in any data being written to the card. I'm also not aware whether .write() can be omitted or bypassed in any way.

I was convinced that
Code: [Select]
datafile.write()
will only move my data to the output buffer on the controller

Correct.

l need
Code: [Select]
datafile.flush()
to actually push it to the card, no?

The buffer is written to the card with .flush(), or .close() or automatically by the library when the 512 byte buffer is actually full.

There are several examples of advanced techniques using multiple buffers, block writes, and binary files in the SdFat library examples files. Continuous high speed data logging without gaps due to the physical write to the card is not simple stuff.

cattledog:
There are several examples of advanced techniques using multiple buffers, block writes, and binary files in the SdFat library examples files. Continuous high speed data logging without gaps due to the physical write to the card is not simple stuff.

Ah, good shout, I'll try using these, but they do seem non-trivial indeed. It'll probably take me a while to figure out whether I can apply anything from there.

Thanks!