Hello everyone. I did some research but had no luck finding any discussion of a similar problem.
While experimenting with ESP32-Cam, I noticed some slow down in writeFile() operation. The more files were already on the SD card the longer the writing took. So I decided to make a test and let it write the following files, each 973 Bytes (32’768 Bytes on the SD card):
1.txt
2.txt
...
40000.txt
I read millis() before and after the write operation. The relevant lines of code are:
// in setup():
SD_MMC.begin()
// in loop():
file_name = counter;
File file = SD_MMC.open(file_name.c_str());
if(!file)
{
write_to_sd_card_start_millis = millis();
writeFile(SD_MMC, file_name.c_str(), data_string.c_str() )
write_to_sd_card_duration_millis = millis() - write_to_sd_card_start_millis;
}
The included libraries are:
#include "FS.h"
#include "SD_MMC.h" // SD Card ESP32
#include "SD.h"
#include <EEPROM.h> // read and write from flash memory
#include "soc/rtc_cntl_reg.h" // Disable brownout problems
#include "driver/rtc_io.h" // Deepsleep IO preparation
The SD card is an old 4 GB card with FAT 32 but other SD cards show similar write times.
The writing takes from ~200ms (no files) to ~3'500ms (40'000 files) regardless of whether the files are in the root directory or not. I could move the already written files to a new folder and writing would still take ~3'500ms. Here is the linear graph:
Any advice would be greatly appreciated. Thank you!
I think it may just be because the library has to read in all the directory entries before it knows whether the file already exists or it needs to create a new one. And a subdirectory pointer entry is just another directory entry with a flag turned on, so it has to go hunting for that too.
But it seems if you low-level format the card to FAT32, then create the subdirectory first, then create all those other files, and finally create a new file in the subdirectory, it seems it should do the final step pretty quickly. It would find the subdirectory entry right away, and determine that it is empty right away, and immediately create the new file. If it still takes a long time, then I'm probably wrong.
Remember that the card itself knows nothing about folders or files. All of that is controlled by your library. A computer could read in the entire folder structure of the card, and know pretty quickly which sector it needs to go to. But your ESP8266 may not have enough ram to do that, so your library has to read stuff in one sector at a time, trying to find a match for the filename.
I'm facing the same issue, I have ESP32Cam saving an image every 30minutes, between captures it sleeps reducing battery usage. After some months it captured over 10,000 pictures and the time it takes to save a picture is now above 5 seconds compared to 150ms when it was empty.
My question is, there is better suited filesystem for this kind of usage where you always write new files?
I was even thinking on using the sdcard without filesystem, splitting it on chunks of 256KB and write each image in one like a circular buffer, I already have a counter stored using Preferences (ESP32 kind of EEPROM), but having a filesystem is more handy.
Possible if your pictures are ALL the same size.
There is an example in the SdFat.h library called LowLatencyLogger that you might want to look at. It creates a series of 128MB files on the card ahead of time, with the directory entries and FAT table entries all done. This has to be done with a freshly formatted card so the successive clusters of the data areas of the files are all sequential. Then it low-level erases all of the data areas.
From that point it writes to successive card sectors directly, without messing with the file system at all, beginning with the sector that marks the beginning of the file (obtained from the directory entry). The card will let you write sequentially until it's full, so you don't even have to update a pointer. But you have to keep track of when to switch to the next file. I don't think there's anything magic about 128MB, so maybe you could just have one file the size of the card.
Then you would need a program or script in your computer that would read in the contents of each file and create a new file for each picture. Of course there would have to be a way to determine when one picture ends and the next begins, but they wouldn't all need to be the same size.
This example was intended to speed up data logging, but it seems the method would also equalize write times for all pictures no matter how many have already been saved. But the parsing script would be an additional requirement.
Thanks ShermanP that's really interesting, I have seen there are some topics here using it, I will definitively take a look to see if it fits my needs
