SDFat is sooo sloooow !!! How to speed-up?

Sorry for bothering again with performances... :wink:
I'm trying to create a set of directories with the following command :

  strlcpy (my_name,"/BLCK/9/2/4/1/2/1/6/5/7/9/8/0/2/6/5/8/1/3",NAME_MAX_SIZE);
  SD.mkdir(my_name);

Guess what, execution of the previous 2 lines of code takes 8.45 seconds (!!!) on an Arduino MEGA running at 16 MHz...
I have initialized SDFat with :

SD.begin(PIN_SDCARD_SSN,SPI_FULL_SPEED)

Is there any way to speed-up the directory creation?
Many thanks!

Are you actually using SDFat.h? Or SD.h?

http://code.google.com/p/fat16lib/

I’m using SDFat, yes. Forgot to say that this is at the beginning of my code :

#include <SdFat.h>
SdFat SD;

BTW, I tried to reformat the card using the tip found at the top of the storage forum... but it doesn't help. :frowning:

fat16lib can correct me if I'm wrong, but to traverse a pathname that is nested 19 levels deep requires a lot of reading of directory blocks which takes up a lot of time. Why on earth do you have such deeply nested directories?

Pete

Yes I understand that it means many operations, and many write accesses to the SD card (which by nature are slow).
I just wonder whether there would be a faster solution.

This structure is a data tree : Tree (data structure) - Wikipedia ... very useful to find information quickly in a vast amount of data (access time is a log function of the number of levels, instead of linear in most other memory structures).

Is there any way to speed-up the directory creation?

If the intermediate directories do not exist this will take a long time. Creation of a directory on a FAT volume requires zeroing a cluster which is likely to be 64 blocks. You have about 20 levels so this will require writing about 1200 blocks.

Access to the FAT and writing directory entries takes more time.

Finally the random access pattern to the SD card will require lots of flash erase delays.

PCs, Macs, and phones have lots of buffer and can cache blocks so you don't have this delay, often 50-100 ms.

The Arduino can only cache a single block so it is very slow for file system operations like the above. Many file system blocks will be read, updated, and written multiple times for the above operation.

Directory operations will also be slow if you have lots of entries in a directory since directories are not indexed.

Thanks fat16lib.
I can't know whether intermediate directories will exist (it will depend on the data being stored in the tree).

I know there would not be more than 10 branches at each level: can this be used to create smaller directories?
Is there a formatting option or a new function to write that would help in this process?

Thanks again!

I know there would not be more than 10 branches at each level: can this be used to create smaller directories?
Is there a formatting option or a new function to write that would help in this process?

The file system on SD cards is designed to match the flash chips used in the SD. Modern SD cards have huge flash erase groups so using smaller clusters won't help much and sometimes makes matters worse.

When you do a random write to a block on an SD card, the SD flash controller may move up to sixteen existing blocks to a new erased flash block together with your new block. This take a long time.

When you do sequential writes to an SD, the SD has enough RAM buffer to avoid flash erases and data movement.

In short your tree idea which goes back to the beginning of time is not practical on Arduino with an SD card.

OK, I see... Looks like we lost real control of the hardware some time ago... :grin:

Thanks anyway for the good tutorial! :slight_smile:

PS : the "tree idea which goes back to the beginning of time " is still in use (check the cache directory of Quicktime downloads, for example)... but it may betray my age! XD

BTW, any chance to at least speed-up the listing?

BTW, any chance to at least speed-up the listing?

Just open the directory file and read it. It is just like any other file (read only) with 32 byte dir_t structs.

PS : the "tree idea which goes back to the beginning of time " is still in use

I know. Classic data structures will always have there place when the storage device can support them.

I have used FRAM chips on Arduino for random access storage. These are probably too small, 256 KB, to help you. FRAM chips are byte addressable and great for linked lists.

Just open the directory file and read it. It is just like any other file (read only) with 32 byte dir_t structs.

Yes, I found the following code, but it looks awkward to open the file to just read its name… Isn’t there another method?

  SdFile file;
  boolean tree_path[MAX_NUMBER_SIZE][10] = {false};  // Content of all levels of the tree leading to the current path
  char tmp_dir[SD_FILENAME_LENGTH] = {0};    // File names are DOS-type 8.3
  int pos = 0;
  int level = 0;

[...]
      level_empty = true;
      while (file.openNext(SD.vwd(), O_READ)) {
        file.getFilename(tmp_dir);
        file.close();
        pos = tmp_dir[0]-'0';
        if ((pos >= 0) && (pos <= 9)) {      // Check filename validity
          tree_path[level][pos] = true;    // Remember available paths at each level in our path
          level_empty = false;
        }
      }
[...]

PS : Yes, FRAM looks extremely promising for embedded… But it’s hard to replace Flash price and density!!

Here is example code to replace your while loop.

I ran it with an three folders in the root directory, the folders were 1, 4, and 9.

#include <SdFat.h>
SdFat sd;
uint8_t sdChipSelect = SS;

bool tree_path[10][10];
void example() {
  int pos = 0;
  int level = 0;
  dir_t dir;
  for (pos = 0; pos < 10; pos++) tree_path[level][pos] = false;

//---------------------------------------
  bool level_empty = true;
  sd.vwd()->rewind();
  while (sd.vwd()->read(&dir, sizeof(dir)) == sizeof(dir)) {
    // done if never used
    if (dir.name[0] == DIR_NAME_FREE) break;
    // skip deleted entry and entries for . and  ..
    if (dir.name[0] == DIR_NAME_DELETED || dir.name[0] == '.'
      || !DIR_IS_FILE_OR_SUBDIR(&dir)) continue;  
    pos = dir.name[0] - '0';
    Serial.write(dir.name, sizeof(dir.name));
    Serial.println();
    if (pos >= 0 && pos <= 9) {
      tree_path[level][pos] = true;
      level_empty = false;
    }
  }
  //----------------------------------

  for (pos = 0; pos < 10; pos++) {
    Serial.print(tree_path[0][pos]);
  }
  Serial.println();
}
void setup() {
  Serial.begin(9600);
  Serial.println("Type any char to run example");
  while (!Serial.available());
  if (!sd.begin(sdChipSelect))sd.errorHalt();
  example();
}
void loop() {}

it prints:

Type any char to run example
1
4
9
0100100001

You may want to replace

      || !DIR_IS_FILE_OR_SUBDIR(&dir)) continue;

with

      || !DIR_IS_SUBDIR(&dir)) continue;

then only directory files will be checked.

Wow !! Thanks !
I knew that reading files one by one was awkward, but I didn't know how to read the directory all at once. Excellent!!!

I tried your loop and it saves ~ 33% of runtime compared to my previous version! XD XD
Thanks again!

PS : Yes, FRAM looks extremely promising for embedded... But it's hard to replace Flash price and density!

Random access to FRAM can be 1000 times faster than to an SD, microseconds vs millisecond. Some users are combining FRAM and an SD to achieve far better performance than you will achieve with only an SD.

256 KB could provide much of your tree structure. A single 256 KB chip is about $7.50.

This is really amazing indeed. It could be interesting to build a cache, and maybe also add a DMA controller to handle the accesses and boost performances.
Interesting as well is the endurance of the FRAM (10^14 write cycles) compared to Flash. No more data moving around to prevent destruction of memory cells!!! XD