Go Down

Topic: Very slow enum files with SD Library (Read 268 times) previous topic - next topic

DrDooom

Jan 23, 2019, 07:48 pm Last Edit: Jan 23, 2019, 07:53 pm by DrDooom
Hello everybody,

I use an ESP32 for the current project and must list files on the SD card in one place. However, that is very slow. I need about ~100 ms to ~120 ms "per file".

So about 12 seconds for 100 files.

Can this be normal? What do you have for times? I test with the following code:
Code: [Select]

...
#include <SPI.h>
#include <SD.h>
const uint8_t PIN_SD_CARD_CS    = 5;
...
  Serial.println(" - SD Card");
  Serial.print  ("   init..."); while (!SD.begin(PIN_SD_CARD_CS, SPI, 10 * 1000 * 1000, "/sd")) { Serial.print("."); delay(1000); }
  Serial.println("done");//SPI 16000000 Baud -> SD Card Read -> ~700kb/sec (5 MB File) / ~8000kb/sec (peeked 512 byte block)
...

        File dir = SD.open("/mp3s");
        uint16_t totalFileCount = 0;
        if(dir){
          dir.rewindDirectory();
          File entry;
          uint32_t enum_files_start = millis();
          while((entry = dir.openNextFile())){
            if(!entry.isDirectory()){ Serial.println(entry.name()); totalFileCount++; }
            entry.close();
          }
          uint32_t enum_files_end = millis();
          Serial.println("enum files duration " + (String)(enum_files_end - enum_files_start) + " ms");
          Serial.println("total file count " + (String)totalFileCount);
        }
        dir.close();


PS: on the ESP32 I have to use "/mp3s" to specify files and directories. There seems to be a wrapper for the SD class expecting this.

Edit: With 20 Mhz SPI Speed (not only 10 Mhz) its a bit faster with ~85ms per file. But still too slow.  :(

Juraj

You can't write an Arduino sketch if you didn't learn programming. Not the language, but the concepts of programming - algorithms and data types.

DrDooom

115200.

And i have commented everything Inside the Loop. Serial and name() is not the problem. Reading Files is possible with 900kb/sec.

DrDooom

I have some Arduino Pro Mini here. I have done a few tests with the small 3.3V 8Mhz.
Just to check the behavior with a normal Arduino (not ESP32).


And ..... the result is very interesting. The Arduino Pro Mini behaves exactly like the ESP32. Sure, it is much slower .... but the times for listing the files are "relative" proportionally identical.

Each listing of a file slows down the next listing.
The first file can be retrieved quite quickly with openNextFile (). the second already a bit slower .... the third even slower ..... etc.

I read on the internet that some people blame the garbage collector. Anyway, it does not seem like a good idea to list many files.

Here are my measurements.

[files attached]

DrDooom

Found my solution: switch to SDfat library. It can list 99 files in ~112ms. (SD library 11994 ms)!

Lucario448

Found my solution: switch to SDfat library. It can list 99 files in ~112ms. (SD library 11994 ms)!
Well, looks like SdFat is the updated version of the stock SD library (and it's weird that, in the new releases of the IDE, the first one is still not included instead of the old one; even when all the necessary wrappers are implemented to maintain compatibility with already existing sketches, except for name()).


Since you said that it takes longer the further it goes, makes me think two things:

  • Card formatted as FAT32 (only for the root directory is a reason, otherwise it's the same as FAT16).
  • It's like every time you call openNextFile(), rewindDirectory() is called as well; which (as you can see) is very inefficient.




I read on the internet that some people blame the garbage collector.
Nonsense, even an ESP isn't powerful enough to afford a garbage collector.
Although some sort of dynamic memory management do exist, it's far away from being a gc. When triggered, a real gc in an Arduino or ESP may "freeze" the execution of the program (including interrupts to avoid race condition) from nearly one to a few seconds; which is not desirable in 99% of the cases.

Go Up