Read directory contents into array

Using the SdFat library, is there a way to read the contents of a folder and look for files that end with a ".dat" and put them into an array (to be used later)? The amount of files in that folder is unknown, but not more than 50.

The idea here is that I don't know how many files are on a card. I need to find only those that end with ".dat" and put them in an array that I can later loop through, opening and reading each one of those files sequentially. The amount of ".dat" files will never exceed 50 (or if it does, I need to truncate at no more than 50 files.)

Hopefully once I learn how to do this, I can expand it into reading folders containing those files as well.

Or, does it make more sense to keep a text file that contains all the file names in it and read that instead?

There is a very old example included with SdFat that demonstrates how to read the contents of a folder and find files.

The example is SdFatTail.ino and is in the folder SdFatBeta20130621/extra/examplesV1/SdFatTail for the latest SdFat beta. The latest beta is here Google Code Archive - Long-term storage for Google Code Project Hosting..

This example finds and prints the tail of files of the form "APPEND.TXT", "PRINT*.TXT", and "WRITE*.TXT".

Here is the example:

/*
 * This sketch reads and prints the tail of all files
 * created by SdFatAppend.pde, SdFatPrint.pde, and
 * SdFatWrite.pde.
 */
#include <SdFat.h>
#include <SdFatUtil.h> // use functions to print strings from flash memory

const uint8_t SD_CHIP_SELECT = SS;

SdFat sd;

SdFile file;

// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))

void setup(void) {
  Serial.begin(9600);
  while (!Serial) {}  // wait for Leonardo
  PgmPrintln("Type any character to start");
  while (Serial.read() <= 0) {}
  delay(200);  // Catch Due reset problem
  
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!sd.begin(SD_CHIP_SELECT, SPI_HALF_SPEED)) sd.initErrorHalt();
  
  // start at beginning of root directory
  sd.vwd()->rewind();
}
/*
 * Print tail of all SdFat example files
 */
void loop(void) {
  dir_t dir;
  char name[13];
  
  // read next directory entry in current working directory
  if (sd.vwd()->readDir(&dir) != sizeof(dir)) {
    Serial.println("End of Directory");
    while(1);
  }
  
  // check for file name "APPEND.TXT", "PRINT*.TXT"
  // or "WRITE*.TXT"
  // first 8 bytes are blank filled name
  // last three bytes are blank filled extension
  if ((strncmp((char *)dir.name, "APPEND  ", 8) &&
      strncmp((char *)dir.name, "PRINT", 5) &&
      strncmp((char *)dir.name, "WRITE", 5)) ||
      strncmp((char *)&dir.name[8], "TXT", 3)) {
        return;
  }
  // format file name
  SdFile::dirName(dir, name);
  
  // remember position in directory
  uint32_t pos = sd.vwd()->curPosition();
  
  // open file
  if (!file.open(name, O_READ)) error("file.open failed");
  
  // restore root position
  if (!sd.vwd()->seekSet(pos)) error("seekSet failed");
  
  // print file name message
  Serial.print("Tail of: ");
  Serial.println(name);
  
  // position to tail of file
  if (file.fileSize() > 100) {
    if (!file.seekSet(file.fileSize() - 100)) error("file.seekSet failed");
  }
  int16_t c;
  // find end of line  
  while ((c = file.read()) > 0 && c != '\n');
  
  // print rest of file
  while ((c = file.read()) > 0) Serial.write((char)c);
  file.close();
  Serial.println();
}

Here is typical output:

Type any character to start
Tail of: PRINT00.TXT
line 96 millis = 2200
line 97 millis = 2201
line 98 millis = 2202
line 99 millis = 2203

Tail of: PRINT01.TXT
line 96 millis = 1312
line 97 millis = 1313
line 98 millis = 1315
line 99 millis = 1316

Tail of: APPEND.TXT
line 98 of pass 99 millis = 22010
line 99 of pass 99 millis = 22012

End of Directory

fat16lib:

  // check for file name "APPEND.TXT", "PRINT*.TXT"

// or "WRITE*.TXT"
  // first 8 bytes are blank filled name
  // last three bytes are blank filled extension
  if ((strncmp((char *)dir.name, "APPEND  ", 8) &&
      strncmp((char *)dir.name, "PRINT", 5) &&
      strncmp((char *)dir.name, "WRITE", 5)) ||
      strncmp((char *)&dir.name[8], "TXT", 3)) {
        return;
  }

Could you explain to me the difference between using 'dir' and '&dir' like you did above ? I'm still trying to learn the different conventions within C/C++ when it comes to variables.

You really need to read a tutorial on C structs, pointers, the address operator '&', the dereference operator '*', and casts.

Here is a start on why I used dir and &dir.

the first two arguments of strncmp must be the address of a C string, an array of type char.

dir.name is the address of an array of uint8_t so (char*)dir.name casts this address to a C string.

dir.name[8] returns the value of the the ninth element of the array name in the struct dir. strncmp needs the address of this element.

The address operator '&' does the conversion so (char*)&dir.name[8] becomes the address of the file extension.

Perhaps someone else can make this clearer. I have programmed in C since the mid 1970s so I don't think about the strange nature of C any more.

Hey thanks for the pointers on what specifically to read up on. I'm slowly understanding the casting of types and when to do that, structs, and pointers. Slow progress.

Also thanks for the code snippet. I think I can figure it out from there. The only question I have is what makes more sense to do, should I:
a) read-as-I-go, as in, read the directory, find a file, open and read, when I'm done, close and move on to the next file and repeat, or
b) read the whole directory and collect all the files in an array and go from there, or
c) keep a separate file, let's call it a "control file" that contains all the data files I need to read and use?

From an 'access' point of view, I don't know what works best as far as accessing the card. Ultimately, each file that gets opened will be constantly read over and over and over again (it's the binary data for each image in my POV system.) After a certain time has elapsed, I pick a new image and display that. Lather, rinse, repeat.

Any suggestions?

This sketch shows how I would do it. I would find the files and use their file index to open them.

The sketch finds all *.TXT files and saves the file's index. It opens them in another for loop.

This is the most efficient way to open a file since the index gives the location of the file's directory entry.

The file index is stored in the array fileIndex[].

/*
 * This sketch demonstrates open by index
 */
#include <SdFat.h>
#include <SdFatUtil.h> // use functions to print strings from flash memory

const uint8_t SD_CHIP_SELECT = SS;

const size_t MAX_FILE_COUNT = 50;

const char* FILE_EXT = "TXT";

SdFat sd;

SdFile file;

size_t fileCount = 0;
uint16_t fileIndex[MAX_FILE_COUNT];

// store error strings in flash to save RAM
#define error(s) sd.errorHalt_P(PSTR(s))

void setup(void) {
  dir_t dir;
  char name[13];
  
  Serial.begin(9600);
  while (!Serial) {}  // wait for Leonardo
  PgmPrintln("Type any character to start");
  while (Serial.read() <= 0) {}
  delay(200);  // Catch Due reset problem
  
  // initialize the SD card at SPI_HALF_SPEED to avoid bus errors with
  // breadboards.  use SPI_FULL_SPEED for better performance.
  if (!sd.begin(SD_CHIP_SELECT, SPI_HALF_SPEED)) sd.initErrorHalt();
  
  // start at beginning of root directory
  sd.vwd()->rewind();
  
  // find files
  while (sd.vwd()->readDir(&dir) == sizeof(dir)) {
    if (strncmp((char*)&dir.name[8], FILE_EXT, 3)) continue;
    if (fileCount >= MAX_FILE_COUNT) error("Too many files");
    fileIndex[fileCount++] = sd.vwd()->curPosition()/sizeof(dir) - 1;
  }
  
  PgmPrint("Found count: ");
  Serial.println(fileCount);
  
  for (size_t i = 0; i < fileCount; i++) {
    if (!file.open(sd.vwd(), fileIndex[i], O_READ)) error("open failed");
    file.getFilename(name);
    PgmPrint("Opened: ");
    Serial.println(name);
    // process file here
    file.close();
  }
  PgmPrintln("Done");
}
void loop(void) {
}

Here is sample output:

Type any character to start
Found count: 3
Opened: DOC.TXT
Opened: TEST.TXT
Opened: ANOTHER.TXT
Done

Ok, that all makes perfect sense. I'll have to move some of that code into the loop() but that's just details. The core of it remains the same. And with the file indexes, I presume that's less information to keep stored in an array as opposed to the entire file name.

Question: is there a wiki or other documentation somewhere that explains all the various pieces of SdFat? Things like when to use SdFat, Sd2Card, SdFile, SdVolume, etc., etc. I think part of my problem earlier was trying to figure out which one gave me the ls() function just to see what's on the card. It would be nice if there was some explanation of all the pieces, including the examples explained in more details. I'm not talking about all of the various calls that one can make within the library like what I'm seeing here, but more of the code like what you wrote in your examples.

For example, I didn't know about using SdFatUtil.h for specific things. There are times I see .begin() and other times I see .init() and I have yet to figure out when to use which. .vwd() ... the "volume working directory" ... ok, is that for any folder I happen to be in, or just for the root folder?

Mind you, this may exist somewhere, and if that's the case, I apologize for not having found it yet.

1 Like

Another question: if I'm only storing the file indexes, does that mean I can't sort them so that I get a list that's either numerically or alphabetically sorted?

Another question: if I'm only storing the file indexes, does that mean I can't sort them so that I get a list that's either numerically or alphabetically sorted?

True. You could store another array of names, and sort the name array, swapping entries in the position array whenever names need to be swapped in the name array.

This is where I wish I could store arrays in arrays. Something like array = {{idx, filename}, {idz, filename}, {idz, filename}}. At least I know how to sort that by specifying the key to use. Alright, I'll keep moving and hope to figure it out.

You need to learn more C. The first version of C had structs in 1973. Structs are special classes in C++ with public members.

Here is a C++ struct that stores the index and name.

struct FileInfo {
  uint16_t index;
  char name[13];  // 8.3 name plus zero byte
};

Here is an array of FileInfo objects:

FileInfo array[50];

Hope you're using a Mega since this approach requires 750 bytes of RAM for the array and a Uno only has 2KB total RAM. Each element of the array requires 15 bytes, two for index and 13 for name.

Why is the order of the files important and why must it be in alpha order? Why do you need to store the names?

There are programs that sort directory entries on FAT volumes. See http://www.murraymoffatt.com/software-problem-0010.html.

I have used the script here http://www.murraymoffatt.com/sortfolder.zip.

The files will be found in alpha order on Arduino after sorting the directory entries in a SD folder.

fat16lib:
You need to learn more C.

Working on that. Having a full time job, managing a performance group, teaching, and being a music instructor doesn't leave a whole lot of time to do anything else. So yeah, it's a slow and painful process.

fat16lib:
Why is the order of the files important and why must it be in alpha order? Why do you need to store the names?

The idea is that the POV display gets synced to music and changes to specific patterns through a piece. I can't simply highlight all the files and copy them to the card, they won't be in the specific order they need to be displayed. I tried, it didn't work. So, I'm either copying each file, one at a time, so they are in order, or I find a way to sort them when I read the card back in. Or, alternatively, I keep a "control" file that lists the data file names in the order in which they need to be displayed, then read that and open the files as needed. This last method is starting to look better and better the more I think about it.

fat16lib:
There are programs that sort directory entries on FAT volumes. See http://www.murraymoffatt.com/software-problem-0010.html.

I have used the script here http://www.murraymoffatt.com/sortfolder.zip.

The files will be found in alpha order on Arduino after sorting the directory entries in a SD folder.

Yeah, that's a possibility. However I need to make it as easy as possible for others to use. It's not just me. The end user needs to be able to simply copy the data files on the card and not have to think about needing to sort them first for it to work right.

alternatively, I keep a "control" file that lists the data file names in the order in which they need to be displayed, then read that and open the files as needed.

That sounds like the best way now that I understand your application. If the time to open a data file is not important you could have the control file open and read the file name then open the file.

If time to open the file is important, read the control file first and make a file index array. It is much faster to open a file by index since open just does a seek to the position of the directory entry.

Open by name rewinds the directory file and does a linear text search to find the directory entry.

That, I don't really know whether it will affect it (by much). The plan with a control file would be, on power up:

  • open control file and get first data file name
  • start timer (this varies, could be 5 seconds, could be 20 seconds)
  • open data file
  • perform multiple, repeated reads of 144 byte chunks
  • loop reading till timer expires (basically display the same image over and over again till the times expires)
  • when timer expires
  • close data file, get next data file from control file
  • open next data file
  • start reading
  • repeat ad infinitum or ad nauseum ...

I'm assuming I can keep the control file open the whole time so I'm not wasting time closing and reopening it each time. The data files are the only ones I will open/close ...

Not sure what kind of delays I will get with that, going from one data file to the next ... I'll need to experiment.

Using a control file is very flexible - not only do you get control of the order, you could repeat a sequence file in the control file's list if you wished; or you could put the duration of each sequence in the control file; or you could have multiple control files with different "shows" drawing from the same or different subsets of the sequence files.

Even if you "keep the control file open the whole time", there are limited buffers so there might be some performance hit coming back to read more of it. But as fat16lib says, you could just read the appropriate control file at the start, look up the files indexes in order and store them in an array - very little space used and faster to transition.

Im sorry to bardge in, but this is the closest i have been to solve my own problem. I have searched for a week now, and found this post.

Im also trying to sort files from sd card in alfabeticly order. My reason for this is that all my files contains different possitions for stepmotors. and when the user chooses a file from a touch tft display it would look nicer if they were sortet.

Im really newbee in c, but my approach is learning by doing!

Well my question is if it is possible. I dont understand how a content file would help, unless the file is sortet?

I really hope you can help me out.

PS sorry for my english!