STM32F1103C8T6 + SD | Fast logging with "SdFat.h"

Hello! :slight_smile:

I need to record 10 or 15 variables on a SD card every 5ms, and the saving process can't take more than 3ms. I am using the following code, where I save 10 variables and calculate the time it takes to save it (Serial.println(time_2 - time_1);. The problem is that the duration of writing is very irregular... there are cycles that last 240ms (time.png). 'time.png' shows a record of (Serial.println(time_2 - time_1);. I have also noticed that I always get the same duration chart as the one I have attached...

Does anyone know how to achieve the goal marked in bold?

Hardware:

Wiring:

  • MOSI: PA7
  • MISO: PA6
  • SCK: PA5
  • CS: PB5
#include <SPI.h>
#include "SdFat.h"
#include "FreeStack.h"

// Use first SPI port
SdFat sd1;
SdFile file1;
// SdFatEX sd1;
const uint8_t SD1_CS = PB15;  // chip select for sd1

const uint8_t BUF_DIM = 10;
uint8_t buf[BUF_DIM];
int32_t time_1, time_2;
//------------------------------------------------------------------------------
// print error msg, any SD error codes, and halt.
// store messages in flash
#define errorExit(msg) errorHalt(F(msg))
#define initError(msg) initErrorHalt(F(msg))
//------------------------------------------------------------------------------

void setup() {
  Serial.begin(115200);
  // Wait for USB Serial
  while (!Serial) {
  }
  Serial.println(F("type any character to start"));
  while (!Serial.available()) {
  }
  // initialize the first card
  if (!sd1.begin(SD1_CS, SD_SCK_MHZ(18))) {
    sd1.initError("sd1:");
  }

  if (!file1.open("test.bin", O_RDWR | O_CREAT | O_TRUNC)) {
    sd1.errorExit("file1");
  }
  delay(100);

  for (size_t i = 0; i < sizeof(buf); i++) {
    buf[i] = i;
  }
}

void loop() {
  Serial.println("----------------------------START");

  for (uint16_t i = 0; i < 10000; i++) {
    delay(3);
    time_1 = micros();
    file1.write(buf, sizeof(buf)) != sizeof(buf);
    time_2 = micros();
    Serial.println(time_2 - time_1);
  }

  file1.close();

  Serial.println("----------------------------CLOSE");
  while (1);
}

time.png

time.png

I suggest you look at the LowLatencyLogger example in the SdFat library. It appears to do everything possible to speed up logging. It begins by creating a 128MB file on the card, then erasing the data portion of the file (so the card's controller won't have to pre-erase before writing). The data section of the file is in sequential sectors. So then it writes logging data to the physical sectors of the card beginning at the sector where the file's data section begins (the beginning cluster location is in the directory entry). It uses the multi-sector write command, which causes the card's controller to automatically move to the next sector, so you can just send successive 512-byte blocks to the card without having to specify where they should be saved.

The key to this is to completely divorce the writing process from anything involving the FAT file system. So when you write the blocks, you don't have to update the FAT, or the second copy of the FAT, or the directory entry. All of those things take time. But since you have predefined the entire file, you can just write successive blocks to the physical card. It's only when you quit logging that you adjust all the FAT-related entries to reflect what was actually saved.

On the controller side, you can use as many 512-byte buffers as you have ram for. When a buffer has been filled, you mark it, then start saving to the next buffer. Your main loop looks for buffers marked as full, and writes them to the card.

I don't know if this all works well enough for your 3000Hz requirement, but the comments in the .ino file say it will do 4000Hz on an Arduino Due if the card is fast enough.

I think for SDXC support, you would need to use the SdFat-beta library. And I don't know anything about getting this to work with STM32.

thanks for the answer :slight_smile:
I had forgotten to mention that using the SdFat library I had already managed to implement what I needed in an Arduino Nano (modifying the LowLatencyLogger example you mentioned). The problem is that the code needs 80% of the memory, and I would need a total of 143% of memory fot the complete project... so I decided to use STM32. But this example is not available for STM32... hence my request for help :frowning:

If you have say 10 floating point numbers (4 bytes each), in each set of samples, and you are saving this every 5 ms, that is a data throughput of 8KB/second which sounds very modest.

On the STM32F103C8T6, you have 20KB ram, so can you see a way of writing to one buffer during the time you are filling the other as suggested by @ShermanP ? Each buffer can hold multiple sets of samples.

The time you need to collect each batch of, say, 10 samples is independent of the block write speed of the SD card.

How are you collecting the samples? via an ADC ? and what processing do you do on them prior to storage ?

Writing a small set of test samples to the SD card, then scaling up the times, may not give an accurate picture of the data throughput because some of the overheads may be independent of the data volume.