SD Write speed deterioration as file size increases

Other Hardware Storage
1

Hi,

I've been running an accelerometer and datalogger for a few months now and I've noticed some strange behavior with regards to the data that has been written to the SD card.

My write speed starts off at around 100Hz but slowly as more data is written to the file the write speed ends up at around 20Hz after maybe 5-8hrs of data is recorded.

Now I can guess why this might be happening? Possibly something to do with it having to open find the last line in the txt/csv file, write data, close, repeat. Each time this takes a little longer as there is a new line of text.....maybe?

Can anybody help suggest as way to fix this please?

I supposed I could ask it to write a new file each time the ardunio is powered up, but that is a bit of a work around and not really addressing the issue.

Thanks in advance

I'm using a Nano, with adafruit micro SD module and LIS3DH acceleromter (+ RTC)

#include <SD.h> //SD Card Module
#include <DS3231.h> //RTC
#include <Adafruit_LIS3DH.h> //Accelerometer
#include <dht.h> //Temp Sensor
#include <SPI.h>
#include <Wire.h>
#include <Adafruit_Sensor.h>


//Temp Sensor
#define dataPin 4 // Defines pin number to which the sensor is connected
dht DHT;

//SD Card Module
File sdcard_file;
int CS_pin = 10; // Pin 10 on Arduino Uno


//RTC
DS3231  rtc(SDA, SCL);


//Accelerometer Module
Adafruit_LIS3DH lis = Adafruit_LIS3DH();


//Millis() function
#define Accelerometer_Intival 20 //Accelerometer reading intival in milliseconds
#define Tempertaure_Intival 60000 //Temperature reading intival in milliseconds


unsigned long currentMillis;
unsigned long Acc_Time;
unsigned long Temp_Time;



void setup() {
  Serial.begin(115200);

  Acc_Time = millis(); //initial start time
  Temp_Time = millis(); //initial start time

  rtc.begin(); //Start RCT

  lis.begin(); //Start Accelerometer

  lis.setRange(LIS3DH_RANGE_4_G);   // 2, 4, 8 or 16 G!

 
  //Temp Sensor Setup
  int readData = DHT.read22(dataPin); // DHT22/AM2302

  float temp = DHT.temperature; // Gets the values of the temperature
  float humid = DHT.humidity; // Gets the values of the humidity

 
 
  // SD Card Initialization - called once on initial power up
  if (SD.begin())
   
  sdcard_file = SD.open("data1.txt", FILE_WRITE);
  if (sdcard_file) {
  sdcard_file.println("### NEW EVENT  ###"); //Denotes new event
  sdcard_file.close(); // close the file
  }
}

void loop() {

  currentMillis = millis();
  accel();
  temp();
}

void accel() //Read accelerometer and write to SD card
{
if(currentMillis > Acc_Time + Accelerometer_Intival)
{

//Accelerometer Read (normalised m/s^2)
  sensors_event_t event;
  lis.getEvent(&event);

//Write to SD Card
  sdcard_file = SD.open("data1.txt", FILE_WRITE);
  if (sdcard_file) {   
    sdcard_file.print(rtc.getDateStr());
    sdcard_file.print("\t");   
    sdcard_file.print(rtc.getTimeStr());
    sdcard_file.print("\t");
    sdcard_file.print(event.acceleration.x);
    sdcard_file.print("\t");
    sdcard_file.print(event.acceleration.y);
    sdcard_file.print("\t");
    sdcard_file.println(event.acceleration.z);
    sdcard_file.close(); // close the file

  Acc_Time = currentMillis;
   
  }
 }
}

void temp() //Read Teamp and Humidity and wirte to SD card
{
  if(currentMillis > Temp_Time + Tempertaure_Intival)
  {

//Temp Read
int readData = DHT.read22(dataPin); // DHT22/AM2302

  float temp = DHT.temperature; // Gets the values of the temperature
  float humid = DHT.humidity; // Gets the values of the humidity

//Write to SD Card

  sdcard_file = SD.open("data1.txt", FILE_WRITE);
  if (sdcard_file) {
  sdcard_file.print(rtc.getDateStr());
  sdcard_file.print("\t");
  sdcard_file.print(rtc.getTimeStr());
  sdcard_file.print("\t");
    sdcard_file.print("\t");
      sdcard_file.print("\t");
       sdcard_file.print("\t");
  sdcard_file.print((float)humid, 2);
  sdcard_file.print("\t");
  sdcard_file.println((float)temp, 2);
  sdcard_file.close(); // close the file
  }

 
Temp_Time = currentMillis;
 
}
}
2

Try leaving the file open.

3

Gary,

I also build high speed dataloggers with LIS331 accelerometers running at 400 Hz and I have seen the same issue of SD time drift slowing the overall data collection speed. The opening time of the SD.open("dataFile.txt",FILE_WRITE) function increases slightly, in microseconds?, with each opening of the SD file. I've started searching through the SD library for a possible cause, but I'm new to C++.

I have found a few work arounds. Call SD.open a minimal number of times. Fill the SD card internal buffer before calling dataFile.close() or dataFile.flush() to write data to the Flash memory cells. Format SD cards with SD Association SDformatter, I prefer the overwrite option.

I hope this helps your issue until we have a better solution. I am looking at a FRAM FIFO buffer for future applications.

4

First, as I posted in the other thread, the formatter provided by the SD Association's Overwrite option leaves your card in the worst possible state - completely full, with no unerased blocks. What you should use is either the low-level formatter in your DSLR camera, or install the SDFat library and use the Example "SdFormatter". Either will leave your card completely erased, so that writing can take place without having to erase something first.

SDFat also has an example called AnalogBinLogger which sets up a file on the card, allocates the rest of the card to that file, erases it, and then simply writes logged data to the card in sequential sectors using the multi-sector write command. During these writes, there is no interaction with the FAT file system, so nothing has to be looked up, and nothing in the FAT table or directory needs to be updated. All of that is updated only when you terminate logging. I think the separation of all the FAT overhead from the very simple function of writing consecutive data to consecutive sectors is going to be key to logging performance.

Also key: The example also uses two 512-byte buffers. Data from the sensors is added to the currently active buffer at precise intervals via interrupts. Then back in the main loop when a buffer is full it is written to the card, and the other buffer becomes active. So you will not lose data so long as the write can be completed within the time needed to fill up a buffer. None of the SD writing activity is bothered by the data-gathering interrupts. Here's the log of my 20-second test run using a Nano at the default 5,000 samples per second.

FreeStack: 1135

type:
c - convert file to csv
d - dump data to Serial
e - overrun error details
r - record ADC data

Sample pins: 0 1 2 3 4
ADC bits: 10
ADC clock kHz: 500
Sample Rate: 5000.00
Sample interval usec: 200.0000
Creating new file
Erasing all data
Logging - type any character to stop

Truncating file
File renamed: analog00.bin
Max block write usec: 1004
Record time sec: 20.130
Sample count: 100642
Samples/sec: 4999.60
Overruns: 0
Done

type:
c - convert file to csv
d - dump data to Serial
e - overrun error details
r - record ADC data
5

ShermanP:
SDFat also has an example called AnalogBinLogger which sets up a file on the card, allocates the rest of the card to that file, erases it, and then simply writes logged data to the card in sequential sectors using the multi-sector write command.

Why does the rest of the card need to be erased? The program writes whatever it has to, and updates the file size. Another other data in there should be irrelevant, no?

6

I may be wrong about that. It looks like it preserves any existing files. But for this method to work, the file to be written to must consist of consecutive sectors. So I guess it erases from the beginning of the new file. If previously written files were consecutive, then that should work ok.