Log data to the SD card/SD card speed

[zacgy]

My project is to adjust the sampling rate up to 10k and save data as CSV file to SDcard

Initially I adjusted the sampling rate to 10k and displayed it through the terminal on the arduino. The result of this adjustment was to receive 10K data in 1 second, but when I added the code for Saving data to SDcard, I found that the CSV data in the SDcard saved is only 200-250 data per second. With this problem, I realized later that it was caused by the SDcard's write speed, and would like to ask for advice on how to make the SDcard write 10k of data per second to a file.

[Paul_KD7HB]

The reality is you can't on a consistent basis. The SD card uses the old floppy disk logic and reads and write in blocks of 512 bytes only.
None of the potential ways I can think of will save enough time to help.
Sorry,
Paul

[jremington]

Post your code, using code tags.

Many people make the terrible mistake of opening the file, writing some data, and closing the file, every time through the sampling loop.

Much of the SD card code you can find on the web is written that way, by people who do not know what they are doing.

[J-M-L]

Bad SD cards can take up to 200 ms for a write and SD cards can pause to do flash wear levelling (in this pause they are copying 128K bytes from an frequently used location to a new location. This is slow and happens outside your control.)

So you should have realistic expectations...

How many bytes do you acquire in one sample (in its written SD representation)?

[zacgy]

Here is my code

#include <ESPAsyncWebServer.h>
#include <SPIFFS.h>
#include <Wire.h>
#include "driver/i2s.h"
// Libraries for SD card
#include "FS.h"
#include "SD.h"
#include <SPI.h>
// Libraries to get time from NTP Server
#include <WiFi.h>
#include <NTPClient.h>
#include <WiFiUdp.h> 

volatile int interruptCounter;
int totalInterruptCounter;
 
hw_timer_t * timer = NULL;
portMUX_TYPE timerMux = portMUX_INITIALIZER_UNLOCKED;
 
void IRAM_ATTR onTimer() {
  portENTER_CRITICAL_ISR(&timerMux);
  interruptCounter++;
  portEXIT_CRITICAL_ISR(&timerMux);
 
}

const char* ssid = "I812";
const char* password = "12345678";

// Define CS pin for the SD card module
#define SD_CS 5


String dataMessage;

// Define NTP Client to get time
WiFiUDP ntpUDP;
NTPClient timeClient(ntpUDP);

// Variables to save date and time
String formattedDate;
String dayStamp;
String timeStamp;
String data_max;

// Create AsyncWebServer object on port 80
  AsyncWebServer server(80);

  
  String readMAX9814DATA() {
  int sensorValue = analogRead(35);
  if (isnan(sensorValue)) {
    Serial.println("Failed to read from MAX9814 sensor!");
    return "";
  }
  else {
//    Serial.println(sensorValue);
    return String(sensorValue);
  }
}

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(1000000);
 
  if(!SPIFFS.begin()){
  Serial.println("An Error has occurred while mounting SPIFFS");
  return;
}
 // Connect to Wi-Fi
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
      // Print ESP32 Local IP Address
    Serial.println(WiFi.localIP());

  timer = timerBegin(0, 16, true);
  timerAttachInterrupt(timer, &onTimer, true);
  timerAlarmWrite(timer, 1000, true);
  timerAlarmEnable(timer);
  
  // Route for root / web page
  server.on("/", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send(SPIFFS, "/index.html");   
     });
  server.on("/MAX9814", HTTP_GET, [](AsyncWebServerRequest *request){
    request->send_P(200, "text/plain", readMAX9814DATA().c_str());
  });    
        
  // Start server
  server.begin();

  // Initialize a NTPClient to get time
  timeClient.begin();
  // Set offset time in seconds to adjust for your timezone, for example:
  // GMT +1 = 3600
  // GMT +7 = 25200
  timeClient.setTimeOffset(25200);

    // Initialize SD card
  SD.begin(SD_CS);  
  if(!SD.begin(SD_CS)) {
    Serial.println("Card Mount Failed");
    return;
  }
  uint8_t cardType = SD.cardType();
  if(cardType == CARD_NONE) {
    Serial.println("No SD card attached");
    return;
  }
  Serial.println("Initializing SD card...");
  if (!SD.begin(SD_CS)) {
    Serial.println("ERROR - SD card initialization failed!");
    return;    // init failed
  }
  // If the data.txt file doesn't exist
  // Create a file on the SD card and write the data labels
  File file = SD.open("/MAX9814.csv");
  if(!file) {
    //Serial.println("File doens't exist");
   // Serial.println("Creating file...");
    writeFile(SD, "/MAX9814.csv", "Date, Hour, MAX9814DATA \r\n");
  }
  else {
    Serial.println("File already exists");  
  }
  file.close();
 
 
  
}

// the loop routine runs over and over again forever:
void loop()
{
   if (interruptCounter > 0) {
 
    portENTER_CRITICAL(&timerMux);
    interruptCounter--;
    portEXIT_CRITICAL(&timerMux);
 
    totalInterruptCounter++;
  getReadings();
  getTimeStamp();
  logSDCard();
  }
}

 void getReadings(){
  data_max = String(analogRead(35));
 }

// Function to get date and time from NTPClient
void getTimeStamp() {
  while(!timeClient.update()) {
    timeClient.forceUpdate();
  }
  // The formattedDate comes with the following format:
  // We need to extract date and time
  formattedDate = timeClient.getFormattedDate();
  //Serial.println(formattedDate);

  // Extract date
  int splitT = formattedDate.indexOf("T");
  dayStamp = formattedDate.substring(0, splitT);
  //Serial.println(dayStamp);
  // Extract time
  timeStamp = formattedDate.substring(splitT+1, formattedDate.length()-1);
 // Serial.println(timeStamp);
}
// Write the sensor readings on the SD card
void logSDCard() {
  dataMessage = String(dayStamp) + "," + String(timeStamp) + "," + String(data_max) + "\r\n";
  //Serial.print("Save data: ");
  //Serial.println(dataMessage);
  appendFile(SD, "/MAX9814.csv", dataMessage.c_str());
}
// Write to the SD card (DON'T MODIFY THIS FUNCTION)
void writeFile(fs::FS &fs, const char * path, const char * message) {
  //Serial.printf("Writing file: %s\n", path);

  File file = fs.open(path, FILE_WRITE);
  if(!file) {
    Serial.println("Failed to open file for writing");
    return;
  }
  if(file.print(message)) {
    Serial.println("File written");
  } else {
    Serial.println("Write failed");
  }
  file.close();
}
// Append data to the SD card (DON'T MODIFY THIS FUNCTION)
void appendFile(fs::FS &fs, const char * path, const char * message) {
 // Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if(!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if(file.print(message)) {
    Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

[jremington]

Yes, your code makes the CLASSIC mistake:

}
// Append data to the SD card (DON'T MODIFY THIS FUNCTION)
void appendFile(fs::FS &fs, const char * path, const char * message) {
 // Serial.printf("Appending to file: %s\n", path);

  File file = fs.open(path, FILE_APPEND);
  if(!file) {
    Serial.println("Failed to open file for appending");
    return;
  }
  if(file.print(message)) {
    Serial.println("Message appended");
  } else {
    Serial.println("Append failed");
  }
  file.close();
}

Open the file once for write in setup(), write data until you are all done collecting data, and then close the file. The speed will vastly increase.

And get rid of all those useless Serial.prints.

[ShermanP]

You may want to look at the SdFat library, and in particular at the LowLatencyLogger example. It goes to considerable lengths to get maximum write speed, including setting up and pre-erasing the files on the SD card in advance so writes can take place by writing directly to consecutive sectors, and updating all the file system data only after you're done. It also uses multiple ram buffers to provide for the occasional long write delay. I don't think it can get any faster than this, but the comments at the top of the code only claim 4000 readings per second with a Due.

I suspect your 10,000 readings per second is simply not possible. But I'd suggest you run the example, and see how fast you can make it work with your card and processor. The default sampling rate is once every 2000us, which is 500 Hz, but you can play with that variable and see how fast it will go.

[J-M-L]

so you want to log in ASCII the Date, the time and the sensed MAX9814 data... so that's likely ~30 bytes worth for one sample, opening and closing the file each time (which you should forget as mentioned by @jremington as a terrible mistake)

at 10KHz that would be close to 300 Kbyte / s

if you look the SdFat library the author says he measured

Shared SPI:
write speed       latency
speed             max        min         avg
KB/Sec,           µs         µs          µs
294.45            24944      1398        1737

Dedicated SPI:
write speed       latency
speed             max        min         avg
KB/Sec,           µs         µs          µs
3965.11           16733      110         127

the library gives sample code for logging (see AvrAdcLogger or ExFatLogger)

if you go for Teensy, then you can look at TeensyDmaAdcLogger and TeensySdioLogger

you'll see though that the techniques used are far from simply opening, writing, closing for each sample.

[jremington]

If the data are collected at regularly timed intervals, it is not necessary to store the time and date along with each data sample.

You will have to think about how to handle the short delays when the internal SD card buffer is written out. Most people use double buffering to get around that, which is not implemented in the standard SD libraries.

But I agree with other posters, you are not likely to meet the stated timing requirements of this project, unless you are extremely familiar with the hardware and the programming language.

[claude_j_greengrass]

@ShermanP

You may want to look at the SdFat library, and in particular at the LowLatencyLogger

As ShermanP suggests, check out the LowLatencyLogger. Using a Mega2560, I'm logging 32bytes per record every 8ms and can do so for 30 seconds without any dropout. The data is recorded in binary format and post processed into a CSV file after the recording is completed. That's only about 8k of data per second which might no be enough for you.

The LLL needs to have data records padded to 2^^n e.g. record lengths must be 2,4,8,16,32... bytes long. I'm "wasting" 8 bytes as I only need 24 bytes/record but it has to be padded to 32 bytes. That wasted space may be enough for your requirements. It would be a close fit IMOSHO but it might work.

I tried to push the recording rate to 400hz with the Mega board, but the LLL wouldn't have it. You may have to go to a faster processor if you need a higher data rate. I have a reference for a fast data logger using a Teensy3.2 if you need to go that route.

ps: I hope you are not using an Uno as it will not have sufficient ram for the buffering that the LLL requires. I ran into that problem as soon as I tried to add code for reading a gyro/accel sensor. YMMV