Hi to all, I am trying to understand what is the best way to write data to sd using a NANO 33 BLE connected to a microSD via SPI. I want to improve performance.
My undestanding is that the write() command of the SD library sends data to a buffer inside the SD card. The SD card don't physically write the data until one of the three following conditions:
- is executed the flush() command
- is executed the close() command
- the amount of data received by the SD surpasses the size of the buffer in the SD
Is this correct?
If it is right, my next question is how big is this buffer? i read about 512 bytes..
If all this is correct, in my case I have readings from IMU with a size of 18 bytes (3 acc x (int16_t) + 3 gyro, + 3 mag) so i can do write() them on each cycle for 28 cycles reaching 504 bytes , and then call flush()
this way data are transferred continuously, but effective write is done periodically.
What is the right approach to this situation ?
Thanks
Emilio.
The buffer is in the Arduino SRAM, 512 bytes for each file.
It is not required to call flush(), but it is a good idea as you won't lose much data if the power fails before you close the file.
However, if you cause the library to write less than 512 bytes, which would fill a sector, then when you resume writing, the library will have to write that entire sector again, and may possibly even have to read in the sector, add additional data at the end to fill it up, then write the full sector again. Not to mention that flushing requires updating the directory entry and possibly both copies of the FAT, twice. You can only write 512 bytes of data to the card at a time - not less or more. So flushing and closing is going to slow you down. It's almost always better to open the file, then write data to it as needed, and let the library worry about batching it up into 512-bytes blocks which it writes to the card when the buffer is full. Then close the file when you're done. You may have to flush or close periodically to protect against a power outage, but try to keep that to a minimum.
The card itself knows nothing about files, or the file system, or directories, or even partitions. Your library has to keep track of all that.
Ok, so if i well undersund i can write all times i want my 18 bytes, the library will group the bytes in blocks of 512 bytes (inside nano33) and write them when the block is ready. I have only to call close() at the end or flush() sometimes to be sure that data, until that moment, are fisically written. right ?
I tried to only use write() , without using flush or close, but if i shut down the micro, the file is empty.
You must close the file to update the file pointers.
flush() does that too, but use it only as needed. If the power goes down, you lose at most the data written since the last flush() command was given.
Ok thank you for your responses,
I am having this approach now, trying to optimize things:
My application is an IMU datalogger that must run at 30 Hz (our customer wants it this way), on each cycle i have 18 bytes, 3 axis accel, gyro, mag, and every variable is int16_t, so two bytes.
In each cycle i read the 9 variables, copy them into a 18 byte char array, and write them to the SD in a single write().
Every 256 cycles I have written 4608 bytes that is exactly 9 pachets of 512 bytes. At that moment i launch a flush() command that write phisically data to SD and will found there all sectors fully filled, so no time wasted in reopening of half filled sectors. After this everithing restart again. Am i right ?
At most only the one current block will be written.
Sorry I don't understand,
my project with this approach is working well, if I power the nano33 and collect some minutes of data and then I cut the power, extract the SD and put into a PC, all data till last flush() are there. Using an oscilloscope connected to a pin where i change status on each cycle i can see an almost perfect square wave with frequency 15 Hz. All seems to work correctly, but i only want to understand if can do things better 
Can you explain better?
Thank you so much,
Emilio, Italy
If you want forum members to help with improving code, post the code, using code tags, and explain any points in doubt.
Sure, sorry, here it is:
#include <Arduino.h>
#include <Arduino_LSM9DS1.h>
#include <SPI.h>
#include <SD.h>
// ===================== OBJs ======================
File file;
// ===================== DEFINEs ===================
#define RED 22
#define BLUE 24
#define GREEN 23
#define TIME_PIN 9
// ===================== VARS ======================
// SdFat SD;
const int chip_select = 10;
const float sample_rate = 30.0;
unsigned long periodMicros, lastMicros, lastMicros2;
bool flag_sd_error = false;
char buffer[18];
int16_t values[9];
int save_counter = 0;
int max_counter = 256;
// =================== STRUCTs ====================
struct
{
float ax;
float ay;
float az;
float gx;
float gy;
float gz;
float mx;
float my;
float mz;
} imu;
void setup()
{
// LEDS
pinMode(RED, OUTPUT);
pinMode(BLUE, OUTPUT);
pinMode(GREEN, OUTPUT);
pinMode(LED_PWR, OUTPUT);
digitalWrite(RED, HIGH);
digitalWrite(BLUE, HIGH);
digitalWrite(GREEN, HIGH);
digitalWrite(LED_POWER, HIGH);
// Digital output (clock)
pinMode(TIME_PIN, OUTPUT);
digitalWrite(TIME_PIN, LOW);
// Begin SERIAL
Serial.begin(115200);
// Begin IMU
if (!IMU.begin())
{
Serial.println("Failed to initialize IMU!");
while (1)
;
}
// Begin SD
if (!SD.begin(chip_select))
{
Serial.println("Card failed, or not present");
while (1)
;
}
// Green led flash
digitalWrite(BLUE, LOW); // turn the LED on (HIGH is the voltage level)
delay(1000);
digitalWrite(BLUE, HIGH);
// Remove the file
if (SD.exists("payload.bin"))
{
SD.remove("payload.bin");
}
// Open the file
file = SD.open("payload.bin", FILE_WRITE); // O_WRITE | O_CREAT);
}
void read_imu(void)
{
if (IMU.accelerationAvailable())
{
IMU.readAcceleration(imu.ax, imu.ay, imu.az);
}
if (IMU.gyroscopeAvailable())
{
IMU.readGyroscope(imu.gx, imu.gy, imu.gz);
}
if (IMU.magneticFieldAvailable())
{
IMU.readMagneticField(imu.mx, imu.my, imu.mz);
}
}
void output_imu(void)
{
// IMU:
Serial.print("Imu:");
Serial.print(imu.ax);
Serial.print(",");
Serial.print(imu.ay);
Serial.print(",");
Serial.print(imu.az);
Serial.print(",");
Serial.print(imu.gx);
Serial.print(",");
Serial.print(imu.gy);
Serial.print(",");
Serial.print(imu.gz);
Serial.print(",");
Serial.print(imu.mx);
Serial.print(",");
Serial.print(imu.my);
Serial.print(",");
Serial.println(imu.mz);
}
void write_sd(void)
{
// Cast float --> int16_t
values[0] = imu.ax * 100;
values[1] = imu.ay * 100;
values[2] = imu.az * 100;
values[3] = imu.gx * 100;
values[4] = imu.gy * 100;
values[5] = imu.gz * 100;
values[6] = imu.mx * 100;
values[7] = imu.my * 100;
values[8] = imu.mz * 100;
// Copy values into buffer
for (int i = 0; i < 9; ++i)
{
memcpy(buffer + i * sizeof(int16_t), &values[i], sizeof(int16_t));
}
// ------- Write to SD without flush --------------
if (file)
{
// Write the buffer to the file in a single operation
int tmp = file.write(buffer, sizeof(buffer));
// file.flush(); // flush is executed only every 256 samples
if (tmp > 0)
{
flag_sd_error = false;
}
else
{
flag_sd_error = true;
}
}
else
{
flag_sd_error = true;
}
// -------- time to flush -----------------------------
if (save_counter < max_counter)
{
// update buffer_index
save_counter += 1;
}
else
{
// Flush
file.flush();
// Reset buffer
save_counter = 0;
// Flash
digitalWrite(GREEN, LOW); // turn the LED on (HIGH is the voltage level)
delay(3);
digitalWrite(GREEN, HIGH);
}
}
void out_clock(void)
{
if (digitalRead(TIME_PIN) == LOW)
{
digitalWrite(TIME_PIN, HIGH);
}
else
{
digitalWrite(TIME_PIN, LOW);
}
}
void loop(void)
{
if (micros() - lastMicros >= periodMicros)
{
// Update timer
lastMicros = micros();
// Read IMU
read_imu();
// Output
output_imu();
// Save to SD
write_sd();
// Clock
out_clock();
}
}
At the risk of complicating things beyond usefulness, I should point out that if the speed of data collection is really critical, there is another approach which is found in the SdFat library's "LowLatencyLogger" example. It creates a series of 128MB files in advance, each with contiguous sectors and clusters, with the data portions of the files low-level erased. So if you were to read the card on a computer, you would see 128MB files filled with FFs.
Then beginning with the first data sector of the file (obtained from the directory entry), collected data is simply written directly to successive sectors of the card without making any changes to the directory entry or the FATs. Then when logging is completed, the directory entry and FATs are adjusted to reflect how much was actually written to the file. If power fails at any point, nothing is lost because the directory still shows a 128MB file. You have all the data that was written, but you also have the erased data at the end. You would just extract the written portion to a new file with a hex editor.
This provides the fastest possible data logging to an SD card that an Arduino can do using SPI. Since the data segments of all files are erased in advance, there is no need to erase a segment before writing to it. And no file system entries are changed until the very end, so no time is wasted doing that. And there is no risk of data loss once it has been written to a segment.
But as you will see reading through LowLatencyLogger, this comes with considerable complexity. So it's best to use normal methods unless speed is critical.
Thank you ShermanP for your suggestion, i will examine that approach.
So what do you think about my previous description where i call the flush after 4608 bytes in order to fill exactly 9 blocks of 512 bytes ? is this a reasonable approach ?
I have used that method on both main frame and mini-computers when data is transient and could never be recreated or only at great time and expense. Works well!
I think your approach sounds reasonable, but you will need to test it.
I did not fully understand when is the moment in which data are passed to the SD buffer. It appen each time the library has collected 512 bytes ?
And what appen in the SD, are data written or not ?
If I call flush after a long time, the buffer would be overfilled.. can you explain to the low level mechanism?
As you write data to the file, the library collects it in a buffer. When the buffer reaches 512 bytes, the library sends all 512 bytes to the card as a sector write. So far as I know, the card's controller writes the data to the card immediately, although it may need to do an erase before writing.
So if you flush after 9 sectors have been written, it's possible the last sector will have already been sent to the card before flush is executed, so the only effect the flush would have is to update the file system entries - the file size field in the directory entry, plus both copies of the FAT.
Hi ShermanP,
I want share with you some tests i have made with my code.
I added to the log file also the timestamp of each reading, i used for simplicity the micros() value from arduino library, in the future, i will use the real timestamp taken from RTC of the nrf52840 of the nano 33 ble.
In the following images you can see the period of each saving, that as expected, is about 33333 uS (see image 3), because the logger is running at 30 Hz. But as you can see the period is not fixed and varies of about +- 1000 uS (image 3), i don't understand this.
Also, as you can see in images 1 and 2 i have sometime small pauses that can be also 70000 uS , about double of the normal period. These pauses appens every some seconds. These pauses obviously slow down my logged data and i loose about 0.6% of the values during the logging.
I can imagine that these pauses are in some way related to SD writings, but i cannot understund why they are so impredictable.
The flush() command furthermore is executed every 11 blocks of 512 bytes (5632 bytes) so it appens every 8.53 seconds, but it does not seems to be related to the spikes.
NOTE:
As i added 4 more bytes of the timestamp (uint32_t) to each cycle, the least common multiple of 22 and 512 is 5632 bytes.
image1
image2
image3
NEW Update !
I was using the native SD arduino library, i changed it and i am now usind SdFat library, (Adafruit fork by Bill Greiman).
Things have changed a lot ! as you can see below in image 4 now there ane no more random pauses ! And i can see clearly my flush() calls (every 8.5 seconds)
The small flutuations in save time remain, but they are not a real problem (image 5), it remains to me to improve the last issue i have with flush() ...
image 4
image 5
That's the default. The minimum buffer to write burst mode is 34 bytes . I only know that from an example I saw and then an article on SD cards.
Without reading all replies, SD cards have an internal MCU, you deal with the code that runs, not the flash. The one mode I have used makes SD look like a DOS drive. I wrote so much DOS software that it's kind of Homey to me.
1 Like
Here i disabled flush() and i used only one close() at the end of sampling, after 60 seconds.
image 6
