Do you want Long File Names in SdFat?

fat16lib:
I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems. No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.

This sounds like the right way to go.

fat16lib:
SdFat is used in very sophisticated apps on Teensy 3.1 and Due. These boards have lots of memory and developers want more features.

I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.

So l for one would appreciate lfn support in SdFat. :slight_smile:

I currently have no need for long filenames, but there are clearly some circumstances where it would be painful to rename everything on an SD card to 8.3. Given your proposed config option to make it optional, sounds like a worthy addition.

I haven't felt a need for LFN. If it adds significantly to processing overhead or to the code footprint, I'd prefer it be an option, I like SdFat a lot just like it is!

I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.

So l for one would appreciate lfn support in SdFat. :slight_smile:

I would ABSOLUTELY like to see LFNs available. The suggestion to support 7 bit rather than 8 bit sounds like a perfect compromise. Arduino applications need to be able to access files that originated outside of the Arduino environment (Mac, Windows, Linux), whether over Ethernet, BT, or from SD cards.

My own application needs to read files from SD cards that were created on a PC or Mac, so long file names is essential.

I will then need to display the filename on the TFT. Of course handling excessively LFNs would require additional processing for the TFT display, but that's fine as far as I'm concerned.

Any thoughts on when a beta implementation might be available?
Thanks!
Neil

Almost all requests have been for opening existing files so I think I will implement that and wait for more interest in creating lfn files.

Opening existing lfn files is not too difficult. I just need to decide what the API should be. In addition to open for a lfn, I need a function for finding all files and returning both short and long names.

Here is output from the test program below that lists all files in a directory. The file name in the first column is the short name.

THELIN~1.PDF The Linux Programming Interface.pdf
SCOTT-~1.MP3 scott-joplin-peacherine-rag.mp3
DAISY-~1.MP3 daisy-bell.mp3
ITS-A-~1.MP3 its-a-long-long-way-to-tipperary.mp3
SHORT.TXT
lower.txt 8.3 lowercase bits: 18
MIXED.TXT Mixed.TXT
Done

#include <SdFat.h>
SdFat sd;
const uint8_t SD_CS_PIN = SS;
//------------------------------------------------------------------------------
// Implements directory byte 12 for lowercase bits with 8.3 names.
// bit 0X10 means lowercase extension and bit 0X08 lowercase basename
void listLfn(SdBaseFile* dirFile) {
  uint8_t offset[] = {1, 3, 5, 7, 9, 14, 16, 18, 20, 22, 24, 28, 30};
  char name[13];
  char lfn[131];
  bool haveLong = false;
  dir_t dir;
  uint8_t i;
  uint8_t lfnIn = 130;
  uint8_t ndir;
  uint8_t sum;
  uint8_t test;

  dirFile->rewind();
  while (dirFile->read(&dir, 32) == 32) {
    if (DIR_IS_LONG_NAME(&dir)) {
      if (!haveLong) {
        if ((dir.name[0] & 0XE0) != 0X40) continue;
        ndir = dir.name[0] & 0X1F;
        test = dir.creationTimeTenths;
        haveLong = true;
        lfnIn = 130;
        lfn[lfnIn] = 0;
      } else if (dir.name[0] != --ndir || test != dir.creationTimeTenths) {
        haveLong = false;
        continue;
      }
      char *p = (char*)&dir;
      if (lfnIn > 0) {
        lfnIn -= 13;
        for (i = 0; i < 13; i++) {
          lfn[lfnIn + i] = p[offset[i]];
        }
      }
    } else if (DIR_IS_FILE_OR_SUBDIR(&dir) 
      && dir.name[0] != DIR_NAME_DELETED 
      && dir.name[0] != DIR_NAME_FREE) {
      if (haveLong) {
        for (sum = i = 0; i < 11; i++) {
           sum = (((sum & 1) << 7) | ((sum & 0xfe) >> 1)) + dir.name[i];
        }
        if (sum != test || ndir != 1) haveLong = false;
      }
      SdFile::dirName(dir, name);
      if (dir.reservedNT) {
        bool dot = false;
        for (char *p = name; *p; p++) {
          if (*p == '.') {
            dot = true;
            continue;
          }
          if (dot && (dir.reservedNT & 0X8) 
            || !dot && (dir.reservedNT & 0X10)) {
              *p = tolower(*p);
            }
        }
      }
      Serial.print(name);
      if (haveLong) {
        Serial.print("  ");
        Serial.print(lfn + lfnIn);
      }
      if (dir.reservedNT) {
        Serial.print(" 8.3 lowercase bits: ");
        Serial.print(dir.reservedNT, HEX);
      }
      Serial.println();
      haveLong = false;
    } else {
      if (dir.name[0] == DIR_NAME_FREE) return;
      haveLong = false;
    }
  }
}
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);
  if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
  // list files in root directory
  listLfn(sd.vwd());
  Serial.println("Done");
}
void loop() {}

Hi fat16lib
An application for which it is really important to read long names files is an mp3 reader.
Thanks to your help (http://forum.arduino.cc/index.php/topic,171663.0.html) I got a way to do it in my project LaRocola II (http://larocola.net)

In a second step, as I do not want to remove the SD card each time I add some music files (for copying them in my PC), I decide to include a FTP server in my mp3 reader. I had published some versions of my FTP server (Ftp Server on Arduino - Networking, Protocols, and Devices - Arduino Forum). For now, it only works with short name files.

As I do not know how to create long name files, I wrote a version of my program that use a table stored in a file in each directory. That table allow me to convert long names to short names (I have not yet published it). It works, but it is not the perfect solution.

So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.

Jean-Michel

So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.

If I add file creation for long file names to SdFat, it must work on Uno with minimal use of memory. Looks like you are using Due so that is not my first choice for testing.

There are lots of LFN implementations that work on ARM with more RAM. I am starting to develop very memory efficient algorithms suitable for LFN creation on Uno. This is a low priority effort since few users need file creation.

I will first finish open for existing LFN files since that is needed to test for an existing file before creating a new file. I think I can do this with no internal buffering for strings. This is important since an LFN can be 255 characters long.

My next priority will be an openNext() that returns the file name to a user specified buffer. This will allow the user to limit the length of the returned name and save RAM.

Hi fat16lib
Thank you for your answer

There are lots of LFN implementations that work on ARM

So I search on the web. I don't found anything except ChibiOS/RT. And, good news, I discover you worked with this OS on Arduino.
So I have two questions:

1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?

2- Can you tell me what other long file names implementations work on Due?

Perhaps my questions are far from the main subject of your post and it would be better to create a new one?
Jean-Michel

1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?

I can't tell you if you should write functions for Due. I use STM32 boards with ChibiOS/RT and FatFS is well supported so I just use Giovanni's port.

2- Can you tell me what other long file names implementations work on Due?

I only use Due for minimal testing of SdFat. I don't use Due for projects so I don't know what is available. I think Atmel has FatFS ports for SAM3X. I think there may also be a FreeRTOS port.

I will soon release a version of SdFat with some LFN support. I plan to extend this to file creation and deletion in the near future.

Here is an example program that lists the long and short file names in a directory and prints the contents of user selected files. This should be enough for many read-only applications.

// Example use of openNextLFN and open by index.
#include <SdFat.h>
#include <SdFatUtil.h>
const uint8_t SD_CS_PIN = SS;

SdFat sd;
SdBaseFile file;
uint16_t index[10];
uint16_t n = 0;
//------------------------------------------------------------------------------
void setup() {
  char name[50];
  dir_t dir;
  
  Serial.begin(9600);
  Serial.println(F("\r\nTest files: SdFat/examples/LongFileName/testFiles"));
  if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
  Serial.print(F("Free RAM: "));
  Serial.println(FreeRam());
  Serial.println();
  
  // list files in root directory
  sd.vwd()->rewind();
  while (file.openNextLFN(sd.vwd(), name, sizeof(name), O_READ) > 0) {
    if (!file.dirEntry(&dir)) sd.errorHalt(F("dirEntry"));
    file.close();
    
    // Skip directories and hidden files.
    if (DIR_IS_HIDDEN(&dir) || DIR_IS_SUBDIR(&dir)) continue; 
    
    // Save index of file in directory.
    index[n] = sd.vwd()->curPosition()/32 - 1;
    Serial.print(n);
    Serial.write(' ');
    Serial.println(name);
    if (++n == 10) break;
  }
}
//------------------------------------------------------------------------------
void loop() {
  int c;
  while (Serial.read() > 0) {}
  Serial.print(F("\r\nEnter File Number: "));
  while ((c = Serial.read()) < 0) {};
  if (!isdigit(c) || (c -= '0') >= n) {
    Serial.println(F("Invald number"));
    return;
  }
  Serial.println(c);
  if (!file.open(sd.vwd(), index[c], O_READ)) {
    sd.errorHalt(F("open"));
  }
  Serial.println();
  char last;
  while ((c = file.read()) > 0) Serial.write(last = (char)c);

  // Print a new line if needed.
  if (last != '\n') Serial.println();
  file.close();
}

This is the size of the program.

Binary sketch size: 10,712 bytes (of a 32,256 byte maximum)

Here is output from the program. First the program lists the files in the directory. It then asks the user to select a file. The content of the file is then printed.

Free RAM: 1094

0 With Two.dots.txt
1 A long name can be 255 characters.txt
2 LFN,NAME.TXT
3 lower.txt
4 MIXCASE.txt
5 mixed.TXT
6 Not_8_3.txt
7 OK%83.TXT
8 STD_8_3.TXT
9 With Blank.txt

Enter File Number: 1

This is "A long name can be 255 characters.txt"
It is has a typical Long File Name.

Enter File Number: 2

LFN,NAME.TXT is not 8.3 since it has a comma.

Enter File Number: 5

mixed.TXT does not have a Long File Name.

Starting with NT, file names of this form,
have the basename and extension character case
encoded in two bits of the 8.3 directory entry.

Here is the the 8.3 listing of the same files.

Files found (name date time size):
WITHTW~1.TXT 2014-11-11 09:14:56 65
ALONGN~1.TXT 2014-11-11 09:14:56 86
LFN_NA~1.TXT 2014-11-11 09:03:54 45
LOWER.TXT 2014-11-11 09:14:56 186
MIXCASE.TXT 2014-11-11 09:14:56 188
MIXED.TXT 2014-11-11 09:14:56 184
NOT_8_3.TXT 2014-11-11 07:57:28 67
OK%83.TXT 2014-11-11 07:54:26 30
STD_8_3.TXT 2014-11-11 07:59:42 33
WITHBL~1.TXT 2014-11-11 08:59:48 59

I posted a new SdFat beta with the first Long File Name support here GitHub - greiman/SdFat-beta: Beta SdFat for test of new features.

See the LongFileName example:

https://github.com/greiman/SdFat-beta/blob/master/SdFat/examples/LongFileName/LongFileName.ino

NB here, at least as far as using 'duinos, etc. Lots of experience in DAS, DBMSs, some web, other stuff. This LFN post was prescient for me. Building a 'normal' data logging set of Teensy 3.1s (humidity, multiple temps, checking for water incursion, overflows, etc) - home app. Not there yet, but next level will be using PI, Beagle, or Galileo (collectors - have to 'play' first). Anyway, for data logging, I've 'always' used file names formed like:
_yyyy-mm-dd_hh:mm:ss. (obviously on the 'larger' machines).
An example might be: Bldg411_2014-06-03_23:28:42.txt. That way, I can group multiple sets of logs together, sorting them in asc or desc order. Always a form of text for content, sometimes XML, sometimes TSV, CSV (don't do that too much). Content tells me which collection source (intend to use right side MAC for T3), which measurement, UTC (IMHO, should ALWAYS store DAS data in UTC time coords), data value (in raw, or Eng-unit converted if that's all I can get). I use SD to store based on DAS req't time entropy so I don't lose it (hate to lose data, and to get from when file(s) are requested from collector). Collection based on

So re LFN, I'd like to create file(s). Intend to create new file during run once stage. As I said, using Teensy 3.1s. Started with Unos, then on to 2560s (which are fine), then to Teensy 3.1s. If exist (not expected), open with append, else create new file. Any timeframe re creation / open with intent to write / append?

SdFat will have reasonable LFN support. I have totally replaced the SdFat core with my new generic FAT12/FAT16/FAT32 code.

I still have lots of work to do so it will still be a while before I post a beta.

Here is the first successful program that creates a LFN.

#include <SdFat.h>

SdFat sd;

SdFile file;

void setup() {
  Serial.begin(9600);
  while (!Serial) {}  // wait for Leonardo

  if (!sd.begin()) {
    Serial.println("begin failed");
    return;
  }

  file.lfnOpen("This is the first SdFat LFN.txt", O_RDWR | O_CREAT);

  file.println("Hello");

  file.close();
  Serial.println("Done");
}

void loop() {}

It is very small but may grow a little as I find and fix bugs.

Binary sketch size: 10,980 bytes (of a 32,256 byte maximum)

If I add a line to print FreeRam I get:

FreeRam: 1133

And here is the what the file looks like on a PC

H:>dir
Volume in drive H has no label.
Volume Serial Number is 34E7-B103

Directory of H:\

01/01/2000 01:00 AM 7 This is the first SdFat LFN.txt
1 File(s) 7 bytes
0 Dir(s) 1,015,398,400 bytes free

H:>type .

This is the first SdFat LFN.txt

Hello

Or with cygwin utilities for those who don't use Windows:

H:>ls -l
total 16
-rw-r--r-- 1 bill None 7 Jan 1 2000 This is the first SdFat LFN.txt

H:>more *
Hello

If people want basic LFN support in SdFat now, they can take a look at the portions of SdFat that are included in the ErikZalm/Marlin project on GitHub. (For example, search for "longFilename" in this file.) LFN support was added to the code some time ago, and it's definitely something of a hack, but it works. It only supports 8-bit characters. :sunglasses:

As a contributor to Marlin, I know we would rather have formal LFN support and include SdFat as an external instead of needing to include a modified copy of SdFat in Marlin's sketch folder.

The current SdFat beta has a proper implementation of long file names for ASCII characters. 16-bit Unicode is not supported but may be in the future.

The code base that ErikZalm/Marlin uses has been totally replace by a new FAT file system implementation.

Long File Names for SD cards is here.

Long File Names for USB drives is here.

I look forward to trying out the beta and seeing how easily it can be integrated into the Marlin codebase. Last time I tried including SdFat as an external my Arduino software wouldn't see it (...grumble...) so for the time-being I'm just dropping the newer SdFat into the sketch.

Wish us luck! We're doing some housecleaning on Marlin to bring things up to speed and clear out the backlog.

Last time I tried including SdFat as an external my Arduino software wouldn't see it (...grumble...)

Did you include SdFat.h in the "sketch". The Arduino IDE has a perverse way of handling the include path.

I just implemented the SDFat library on an Arduino Ethernet board with no issues.
We wanted a way to save service files inside our TCP connected equipment and your library made it a breeze so thank you very much.

I did this under IDE 1.0.6 and the SPI at full speed. I had no issues at all with your code.

Cheers!

I recently decided to try and build an music player using a hardware decoder IC (VS1053) as a slave and Arduino as the user interface and file system...

I was hoping to copy my files directly from the computer to the SD card or USB with file names in tact... For my part, I tend to restrict file names to 32 characters even on the PC, so limiting long file names is not an issue.

The final project intends to use a decent TFT display with either touch sensor or capacitive keys (not decided) and should for the most part provide a solution similar to my favourite DOS MP3 player... MPXPlay.

While the same could be accomplished with 8.3, it complicates the process, so LFN support would help me greatly... not to mention making the project more agreeable and modern...

Mike