Do someone think there can be a way to speed up the flush() method ?
Buy a faster SD card, and use a faster SPI clock setting.
1 Like
I think SdFat is a better performing library than SD. Bill Greiman went to a lot of trouble to make things work as fast as possible. But with regard to flushes, aside from jremington's suggestions, the only way I know to make them not interfere with regular data logging is to not do them at all - use the Low Latency Logger method. Remember that the flush will rewrite the directory entry and both copies of the FAT. To do that, the card's controller will probably have to do some erasing first. So it just takes time.
1 Like
When SD reads the buffer with enough data to reach and sustain burst speed, that is the efficient use of SD. Hitting the handle every less than 34 bytes actually interrupts any chance of getting out of 1st gear.
WHY do you NEED to FLUSH often?
Hi GoForSmoke, i don't need to flush() often, my application will be used to monitor displacements of a device for a very long period, about two or three months, so i can execute a flush also every hour or more. The problem is that i would like to avoid every single sample loss, because my data will be processed by other companies and i don't want to be asked about sample leaks by other peaple... In the worst scenario i can obviously postprocess my data (i already have to convert them from binary to CSV..) and fix holes by linear interpolation. It can be done simply, as i save togheter with every record also the timestamp, so i can detect every gap and fill it.
I made some measurements with my oscilloscope and it confirmed that execution time of flush() command spans from 60 to 100 ms, as i saw in my logged data above !
I read that this command should take about 10 ms, why is it so long in my code ?
I also changed the frequency of the command, but it is always so long...
Oscilloscope
Can you save reads to onboard or wired EEPROM then write SD between read periods and then check the data filed while you still have the EEPROM image? Uno chip has 1K EEPROM, ATmega1284P has 4K, might cost $6 or so in qty 1.
I have closed files after every record written over not losing data before but only when there was time for it and re-opening. That was in the 80's.
What is the actual risk of data loss? Would it be a power outage, or something else? If it's power, then you might do better adding a battery backup or something like that. Even if you do the flushes, you could still lose up to 512 bytes of data if the power goes out. So it might be best to make sure the power won't go out.
If the power goes out before the file is close, the FAT doesn't get updated and you lose everything since the last close. That's how it's always been with DOS FAT files.
You could also send the data to a laptop terminal emulator and with more work, to your phone.
Hi to all,
my application will be battery-powered and installed in a place that is far from any PC or other devices, it will be used underwater in the ocean.
I am still trying to understand how to improve the flush() command to make it faster, but I don't have any new ideas.
The only solution I can see now is to correct data in post-processing by performing a linear interpolation across the missing data.
You want to make it faster, get the fastest flash you can and run at a high clock rate for that card. Perhaps a 400 to 600 MHz PJRC Teensy would be overkill but might not and at least one has an AMD with FPU.
Burst writes are fastest. Less than burst is slower so dibs and drabs and dinking around trying to fool the card only stooges it up.
The SD card max SPI clock speed, what is that?
AVR-duino default is 4 MHz. With AMD, 100+ MHz and you want speed.
Hi to all,
After some days of hard work 
I finally wrote a code using sdfat library that can save my device's data at nearly 70 Hz !!
I spent some time studying the "LowLatencyLogger" example initially posted by ShermanP and I created a "simplified" version that runs perfectly for my problem.
The approach was the following:
- I created a big "payload.bin" file of 4 GB using Python on my PC containing only 0xFF values
- I copied this file to the SD formatted as FAT32
- In the firmware the first thing I did was to read the first Sector value using the related method of the File Class
- After this I closed the file and terminated the Class sd.
- At this time I went to use the SdCard Class that gives access to low-level write to the sd card, and I used the writeSector() method to write in the loop the 512 byte packets of data collected from the IMU.
- during each cycle I collect data from IMU and when the buffer is full I write it to the next sector of the file. I also added some control bytes to simplify the decoding process on the PC, using a Python app.
The FAT32 structure is not affected at all, and when the SD is inserted to a PC the bin file is seen full as 4 GB, the only difference is that bytes inside have changed.
In this part i read the firstPacket position:
// ================ Begin SD ===================
if (!sd.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(8))))
{
Serial.println("Card failed, or not present");
while (1)
;
}
Serial.println("SD card initialized");
// ================ Open file to read first sector =============
if (!file.open(FILE_NAME, FILE_READ))
{
// Print an error message and halt
Serial.println("Failed to open file.");
while (true)
{
}
}
Serial.println("File opened to read first sector");
// ================ Open file to read first sector =============
first_sector = file.firstSector();
Serial.print("First_sector = ");
Serial.println(first_sector);
// ================ Close the file ========================
file.close();
Serial.println("File closed");
// ================ Close the sd object ========================
sd.end();
Serial.println("SD closed");
In this part I create the "card" object from SdCard Class:
// ==================================================================
// LOW LEVEL ACCESS
// ==================================================================
if (!card.begin(SdSpiConfig(SD_CS_PIN, DEDICATED_SPI, SD_SCK_MHZ(8))))
{
Serial.println("Card initialization failed!");
return;
}
Serial.println("SD Card opened");
// ================ Card details ========================
Serial.print("Card type: ");
switch (card.type())
{
case SD_CARD_TYPE_SD1:
Serial.println("SD1");
break;
case SD_CARD_TYPE_SD2:
Serial.println("SD2");
break;
case SD_CARD_TYPE_SDHC:
Serial.println("SDHC");
break;
default:
Serial.println("Unknown");
}
Serial.print("Card size: ");
Serial.print(card.cardSize() * 512E-9);
Serial.println(" GB");
And finally this function writes the sectors inside the file:
void write_sd(void)
{
// insert timestamp inside the buffer
values[0] = (uint16_t)(timestamp & 0xFFFF); // Extract low word
values[1] = (uint16_t)((timestamp >> 16) & 0xFFFF); // Extract high word
values[2] = imu.ax * 100; // Cast float --> int16_t
values[3] = imu.ay * 100;
values[4] = imu.az * 100;
values[5] = imu.gx * 100;
values[6] = imu.gy * 100;
values[7] = imu.gz * 100;
values[8] = imu.mx * 100;
values[9] = imu.my * 100;
values[10] = imu.mz * 100;
// Copy the values array to the buffer
memcpy(buffer + buffer_counter, values, sizeof(values));
buffer_counter += sizeof(values);
// Check if the buffer is full (506 bytes = 23 samples x 22 bytes)
if (buffer_counter >= 506)
{
// Add the mission id at position 507-510 (4 bytes)
memcpy(buffer + buffer_counter, mission_id, sizeof(mission_id));
// Write to card
digitalWrite(TIME_PIN, HIGH);
if (!card.writeSector(first_sector + 1 + sector_counter, buffer))
{
flag_sd_error = true;
}
digitalWrite(TIME_PIN, LOW);
// Reset buffer size after writing to SD card
buffer_counter = 0;
// Increase sector counter
sector_counter += 1;
// Flash
digitalWrite(GREEN, LOW); // turn the LED on (HIGH is the voltage level)
delay(3);
digitalWrite(GREEN, HIGH);
}
}
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.