file read times depending on directory file count?

Hey. I'm using the following code to open files from an sd card in sequence, read the data, and send it to a dmx device. It's working fine (<20ms between each file opening) with a small (around 100 or less) number of files, but when I increase the number of files the read speed increases dramatically.

What's actually happening now, with 900 files, is the first 10 or so files take ~300 ms each to open, then the next file opens in~70ms, then each consecutive file takes slightly longer to open, until it takes ~500 ms for each file to open.

Can anyone explain why this is happening, and how to get around it?

thanks

//   #define MEGA_SOFT_SPI 1 must be set in 2d3card.h to use dmx shield sd !!!!!

//   DmxSimple.h must be edited to include Arduino.h, not Wiring.h !!!!

#include <SD.h>
#include <DmxSimple.h>



File myFile;

unsigned long millisPre;
unsigned long millisPost;
unsigned long m1;
unsigned long m2;
int testLoopCount;

void setup()
{

  DmxSimple.usePin(2);
  DmxSimple.maxChannel(3);
  Serial.begin(57600);
  Serial.print("Initializing SD card...");
  // On the Ethernet Shield, CS is pin 4. It's set as an output by default.
  // Note that even if it's not used as the CS pin, the hardware SS pin 
  // (10 on most Arduino boards, 53 on the Mega) must be left as an output 
  // or the SD library functions will not work. 
   pinMode(53, OUTPUT);
   
  if (!SD.begin(10)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  

    
  unsigned char testBuff[513];

  
  Serial.println("beginning read");
  millisPre = millis();
  //Serial.println(millisPre);
   
  
  
//  testLoopCount = 1;
//  for(int cycles = 0; cycles < testLoopCount; cycles++)
//  
  
  while(1)
  { 
    static char filename[] = "fr00000#.txt";
    static char filepath[] = "frames/";
    filename[7] = '#';
    filename[6] = '0';
    filename[5] = '0';
    filename[4] = '0';
  
    
     
    for (int thous = '0'; thous <= '9'; thous++)
    {
        filename[4] = thous;
        
        for (int hnds = '0'; hnds <= '9'; hnds++)
        {
            filename[5] = hnds;
            
            for (int tens = '0'; tens <= '9'; tens++)
            {
                filename[6] = tens;
                
                for (int ones = '0'; ones <='9'; ones++)
                {
                  filename[7] = ones;
                  
                  m1 = millis();
             
            
                    // open the file for reading:
                    myFile = SD.open(filename);
                    //myFile = SD.open(strcat(filepath,filename));
                    //myFile = SD.open(filename);
                    if (myFile) 
                    {
                        Serial.println(filename);
                        
                        // read from the file until there's nothing else in it:
                        int byteCount = 1;
                        while (myFile.available()) 
                        {
                            
                            DmxSimple.write(byteCount, (int)myFile.read());
                            //testBuff[byteCount] = myFile.read();
                            //testBuffInt[byteCount] =(int)testBuff[byteCount];
                            byteCount++;  
                     
                        }
                        //testBuff[byteCount] = NULL;
        
                        ///////////////
                       
                        
                        // close the file:
                        myFile.close();
                    } 
                    else 
                    {
                      // if the file didn't open, print an error:
                      //Serial.print("error opening ");
                      //Serial.print(filename);
                      //Serial.println(" test.txt");
                      {goto bailout;}
                    }
                    
                    
                    m2 = millis()-m1;
                    Serial.print(" ");
                    Serial.println(m2);
                    m1 = m2;
                  
                }
                //filename[6]++;
                 
            }
        }
    }
    bailout:;

  }


    
  
   Serial.println();
   millisPost = millis();
   //Serial.println(millisPost);
   Serial.println((millisPost-millisPre)); 
   

   
   
}



void loop()
{
	// nothing happens after setup
}

Opening a file requires a search of the directory file. Each directory entry is 32 byte long with a 11 byte file name. Reading the entry and doing a compare of the name in the directory with the name of the file to be opened takes time.

Directory entries are not in any simple order. When a file is created the first free entry is used. This means a sequential search must be used.

If you started with a freshly formatted SD and created the files in the order you want to access them you could use the SdFat function to open the file by index. This function is very fast but tricky to use.

Here is the function with doxygen documentation comments:

//------------------------------------------------------------------------------
/** Open a file by index.
 *
 * \param[in] dirFile An open SdFat instance for the directory.
 *
 * \param[in] index The \a index of the directory entry for the file to be
 * opened.  The value for \a index is (directory file position)/32.
 *
 * \param[in] oflag Values for \a oflag are constructed by a bitwise-inclusive
 * OR of flags O_READ, O_WRITE, O_TRUNC, and O_SYNC.
 *
 * See open() by path for definition of flags.
 * \return true for success or false for failure.
 */
bool SdBaseFile::open(SdBaseFile* dirFile, uint16_t index, uint8_t oflag);

Yes I can create them in the order to be used. At the moment I'm just using the standard sd library as you can see above. Is there much involved in converting what I've done to fat16lib and using the sequential technique? It sounds perfect for my needs.

You need to use SdFat. The SD.h library is a wrapper for an old version of SdFat. SD.h presents a slightly modified API for SdFat but adds no features.

SdFat is here Google Code Archive - Long-term storage for Google Code Project Hosting. or beta versions are here Google Code Archive - Long-term storage for Google Code Project Hosting..

In addition to open by index, you might be able to use openNext(). OpenNext() should be a fast way to open every file in sequence.

Nice, so I should be able to remove my filename code too.

Does OpenNext() loop when it reaches the last file?

One problem though, I need to cycle through something in the order of 5000 files. As I understand it, sdfat supports only fat16, which only allows 512 files per folder?

Is this irrelevent though if I'm using OpenNext() ? Can I just break up my files into folders with 512 each, and OpenNext() will skip between them all regardless?

You don't want fat16lib which is a smaller FAT16 only library. I also wrote SdFat which is the base for the official Arduino SD.h library. SD.h uses a very old version of SdFat. SdFat supports FAT32 and FAT16.

openNext() does not cycle but you simply rewind the directory to restart. Directories are just files so you can position them with rewind or seek.

Hi fatlib

I've tried calling file.rewind() after openNext has reached the end of the file list but it doesn't return the beginning. What am I doing wrong?

this code should just keep looping through files

while(1)
  {
      // open next file in root.  The volume working directory, vwd, is root
      while (file.openNext(sd.vwd(), O_READ)) {
        file.getFilename(name);
        cout << name << endl;
        file.close();
      }
    
      file.rewind();
  
  }

You need to rewind the directory file.

  sd.vwd()->rewind();

splendid :slight_smile: