SD write speed & sample rate stability - esp32

Hi,

I've been trying to squeeze as much out of my SD datalogger as possible but I just don't seem to be able to get above a 80Hz sample rate using a Nano. I thought maybe if I tried using the esp32 (wroom 32) I'd smash 80Hz, but in fact I'm getting less than 20! I'm clealy missing something or not fully understanding the hardware. Can any help me understand whats going on please?

On paper, at least to my eye, the esp32 is just faster, so why is the write speed so slow?

This is the Nano code I'm using (sdfat)


// Basic demo for accelerometer readings from Adafruit LIS3DH with RTC

#include <Wire.h>
#include<SPI.h>
//#include "FS.h"
#include "SdFat.h"
#include "RTClib.h"
#include <Adafruit_LIS3DH.h>
#include <Adafruit_Sensor.h>

//----------------------

// I2C
Adafruit_LIS3DH lis = Adafruit_LIS3DH();

//SD Card
SdFat SD;
File dataFile;
const uint8_t SD_CHIP_SELECT = 10;

//SD Filename
char fileName[25];

//RTC2321
RTC_DS3231 rtc;
char DateAndTime[20];

//----------------------

//Millis Setup
const unsigned long currentMilis = millis;
const unsigned long newMillis = 0;

//=====================================================================

void setup(void) {

  Serial.begin(9600);

  //LIS3DH Accelerometer
  lis.begin(0x18);
  lis.setRange(LIS3DH_RANGE_4_G);   // 2, 4, 8 or 16 G!

  //RTC
  rtc.begin(); //Start RCT

  //SD Card
  SD.begin();
  SdFile::dateTimeCallback(dateTime); //timestamp file


}

//=====================================================================

void loop() {

  Accel();

}

//=====================================================================

void Accel() {

  DateTime now = rtc.now();
  sprintf(DateAndTime, "%02d/%02d/%02d,%02d:%02d:%02d", now.year(), now.month(), now.day(), now.hour(), now.minute(), now.second());

  getFileName();

  sensors_event_t event;
  lis.getEvent(&event);


  dataFile = SD.open(fileName, FILE_WRITE);

  if (dataFile) {

    dataFile.print(DateAndTime);
    dataFile.print(",");
    dataFile.print(event.acceleration.x);
    dataFile.print(",");
    dataFile.print(event.acceleration.y);
    dataFile.print(",");
    dataFile.println(event.acceleration.z);

    dataFile.close();
  }
}

//=======================================================================

void getFileName() {

  DateTime now = rtc.now();
  sprintf(fileName, "Accel_%02d%02d%02d_%02d%02d.csv", now.year(), now.month(), now.day(), now.hour(), now.minute());
}

//=======================================================================

void dateTime(uint16_t* date, uint16_t* time) {
  //used for callback function for file timestamp

  DateTime now = rtc.now();

  // return date using FAT_DATE macro to format fields
  *date = FAT_DATE(now.year(), now.month(), now.day());

  // return time using FAT_TIME macro to format fields
  *time = FAT_TIME(now.hour(), now.minute(), now.second());
}

This is the esp32 code I'm using (standard SD library as sdfat wouldn't work)

#include <Wire.h>
#include<SPI.h>
#include "FS.h"
#include "SD.h"
#include "RTClib.h"
#include <Adafruit_LIS3DH.h>
#include <Adafruit_Sensor.h>

//----------------------

// I2C
Adafruit_LIS3DH lis = Adafruit_LIS3DH();

//SD Card
#define SD_CS = 5;
File file;

//RTC2321
RTC_DS3231 rtc;
char DateAndTime[20];

//----------------------

String timeStamp;
String accelData;
float accl_X;
float accl_Y;
float accl_Z;


//=====================================================================

void setup() {

  Serial.begin(9600);
  Wire.begin();

  //LIS3DH Accelerometer
  lis.begin(0x18);
  lis.setRange(LIS3DH_RANGE_4_G);   // 2, 4, 8 or 16 G!

  //RTC
  rtc.begin(); //Start RCT

  //SD Card
  SD.begin();
}

//=====================================================================

void loop() {

  readAccelData();
  logSDCard();

}

//=====================================================================

void readAccelData() {

  //First read time

  DateTime now = rtc.now();
  sprintf(DateAndTime, "%02d/%02d/%02d,%02d:%02d:%02d", now.year(), now.month(), now.day(), now.hour(), now.minute(), now.second());

  //Now read accel data

  sensors_event_t event;
  lis.getEvent(&event);

  //reformat infomation

  timeStamp = DateAndTime;
  accl_X = event.acceleration.x;
  accl_Y = event.acceleration.y;
  accl_Z = event.acceleration.z;
}


//=======================================================================

// Write the sensor readings on the SD card
void logSDCard() {

  accelData = String(timeStamp) + "," + String(accl_X) + "," + String(accl_Y) + "," +
              String(accl_Z) + "\r\n";

  appendFile(SD, "/datafile.csv", accelData.c_str());

}

//=======================================================================

void appendFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if (!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if (file.print(message)) {
    Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

Thank you :slight_smile:

Processor speed has little bearing on the data rate of the interface to the SD card.
Paul

Have you tried NOT openeing and appending, with each read? You could open it, do 1000 writes then close and reopen.

2 Likes

ah, so I've up the baud rate to 115200 and I'm getting 100Hz for the first few seconds, then the sample rate starts to reduce quite quickly. I guess this is because it needs to find the last line in the .csv file before it writes. Is there any way of storing say 10kb to a buffer then writing that data to the sd card while another buffer stores the next lot of incoming data? Would that even work?

Thanks

Not yet, I'll give that a go also! Thanks

Doesn't seem to make much difference. Maybe I'm missing something else? I need to think of other things and go back to it fresh I think.

Is the SD card the ONLY device using SPI?
Paul

Yes, everything else is I2C

The issue now that I've increased the baud rate is the reduction in sample rate as the file increases in size (or as the line count increases).

How much reading have you done relating to SD cards? They ONLY read and write in 512 byte blocks, no matter what else you do in the program, exactly like to old floppy disks on ancient PCs.
Paul

Did you follow @missdrew's advice? If not, what you described is what would be expected to happen.

I have not studied your program, but one sure way to speed up SD card processing is to make the individual record fixed length and the length fitting EXACTLY into 512 bytes.
Paul

I have, or at least I think I have, but seem to be getting a similar result.

void loop() {


  for (int i = 0; i < 100; ++i) {

    readAccelData();
    logSDCard();

  }

  file.close();

}

It starts off well, but soon starts to deteriorate as the csv line count increases.

Could you give me more info on what I might be able to do to keep the writes consistent throughout the csv file please? @missdrew @PaulRB

Thank you

This command is still getting executed for each line written.

File file = fs.open(path, FILE_APPEND);

This requires the library to open the file and then read through it to find the end, before the new line can be written. You need to restructure your code so that the file is only opened for append once for each batch of records.

Thank you @PaulRB, I'll look at this today after the kids have worn me out. Appreciate your time.

In case you did not realise:

//SD Card
#define SD_CS = 5;
File file;

You have 'file' as a global variable and...

void appendFile(fs::FS &fs, const char * path, const char * message) {
  Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if (!file) {

... you have a local variable with the same name in that function. These are two different variables! The local variable makes a hole in the scope of the global variable so that inside that function, 'file' refers to the local variable, and elsewhere, 'file' refers to the global variable. The reason that a local variable is created is because you put the type 'File' in front of the variable name 'file'. If you removed 'File' then no new local variable would get created, and 'file' would refer to the global variable.

1 Like

Closing a file can take a lot of time. The directory entry has to be modified to reflect the file length, so that sector has to be read by the library, modified, and re-written back to the card. And the card's controller may have to either erase the block that directory sector is located in, or relocate it in some wear-leveling way. So you want to open the file in the beginning, write to it consecutively, then close it only when you're ready to shut down. One open, one close.

If you want the fastest logging method, take a look at the LowLatencyLogger example in the SdFat library. It creates a 128MB file on the SD card in advance, made up of consecutive sectors, pre-erases the data portion to all FFs so all future writes can be done without having to erase anything first. Then it sets up multiple 512-byte buffers to deal with the occasional slow write, and the actual logging process is just writing data to consecutive SD card sectors in 512-byte chunks. There's no messing with the file system at all until the end - no updating the directory entry, no looking for the next cluster to use, no updating the FAT or the second copy of the FAT. It's only at shutdown that the directory entry and the FATs are updated to reflect what was actually written. You really can't get any faster than this.

1 Like

Hi @ShermanP, thanks for the info. Unfortunately the Sdfat library doesn't support the esp32 :frowning: Maybe in the future it will, one can only hope.

I understand what everyone is saying to me and what I need to do but I'm getting pretty frustrated because each time I try recode it to open, write and then close once I've finished collecting data it either doesn't write at all, or I get the same result. I'm just using a simple millis() (or esp_timer_get_time() for the esp32) to open the SD card capture the data, write the data for a set amount of time, then close the file.

I thought this issue would be covered more thoroughly in general as it must be a common issue considering the amount of basic sd datalogger examples out there.

I'll knock together a state machine later on and see if that helps rather than using the loop.

But you're already using SdFat on the Nano, so you could just try out the LowLatencyLogger example and see what speed you can get.

Yes, you're right. I'll rebuild the nano version this evening and give it a whirl.