Go Down

Topic: Do you want Long File Names in SdFat? (Read 19570 times) previous topic - next topic

fat16lib

I would appreciate comments on the need for LFN, Long File Names, in SdFat.

I have not implemented LFN because of the complexity, amount of flash, and RAM required.  I am now looking at making LFN a configuration option so the default size of SdFat would not increase.

LFN uses UTF-16 characters but I only plan on supporting an 8-bit character set.

LFN is slow and uses lots of memory.
Quote

The long file name system allows a maximum length of 255 UTF-16 characters, including spaces and non-alphanumeric characters.  This is achieved by chaining up to 20 directory entries of 13 2-byte unicode characters each.


Allowing each portion of a path name to be 255 characters long requires lots of memory.  Some implementations of FAT for embedded systems limit the maximum length of a file/directory name.  I plan to provide a configuration parameter so small memory Arduinos like Uno can use limited LFN.

Please comment on on this plan.

Peter_n

Files created by Arduino : LFN not needed.
But I would like to open files created by a computer with a long name.

robtillaart

LFN might make sense on the DUE which has more RAM and processing power.

For the UNO/MEGA scope I agree with Peter,
- reading them would be the first need.
- rename them to an 8.3 file name would be the second need

some questions came up?
are there other file formats for SD cards e.g. those from linux that could be applied to an SD that are faster in general?
Can an SDcard have partitions?

Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

Peter_n

When you think about other file formats, you could do simple binary writes without file system and read and convert it in linux. It is not that hard in linux.

In the past, some USB memory sticks did not have a partition table, just a straight forward FAT16 file system. As far as I know, today most memory sticks/cards have a partition table with a single partition. So you you can have many partitions, just as with a harddisk.

fat16lib

#4
Sep 13, 2014, 07:02 pm Last Edit: Sep 13, 2014, 07:29 pm by fat16lib Reason: 1
Quote

Files created by Arduino : LFN not needed.
But I would like to open files created by a computer with a long name.

This sounds like a good first step.  I already have a prototype working.  The prototype supports 7-bit characters, the original ASCII http://en.wikipedia.org/wiki/ASCII.  Extending to 8-bit SBCS code pages or UTF-16 adds a huge amount of complexity.

The the extra characters in standard U.S. Microsoft code page 437 are now anarchic http://en.wikipedia.org/wiki/Code_page_437.

UTF-16 implementations take over 100 KB of flash http://elm-chan.org/fsw/ff/en/appnote.html.

Quote
are there other file formats for SD cards e.g. those from linux that could be applied to an SD that are faster in general?
Can an SDcard have partitions?

Long File Names do not change the speed of I/O.  Just the time to open/create files due to the directory complexity.

Linux/Unix file systems are way too complex for Arduino and would not be faster on SD cards.  The flash structure of SD cards is designed for FAT file systems.

The fastest way to access a file on an SD is to create a contiguous file and access it using multi-block raw SD reads and writes.  SD cards are fast only if you use very large multi-block transfers since the flash pages are now some multiple of 16 KB.

See the SdFat LowLatencyLogger example for an implementation https://github.com/greiman/SdFat/blob/master/SdFat/examples/LowLatencyLogger/LowLatencyLogger.ino.

Quote
As far as I know, today most memory sticks/cards have a partition table with a single partition. So you you can have many partitions, just as with a harddisk.

The base code in SdFat allows four partitions on an SD card or a single floppy style volume.  The SD standard now only permits a single partition with a MBR so this feature is hidden.


robtillaart

Thanks for the answers, informative...
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

CatweazleNZ

Hi

If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.

Cheers

Catweazle NZ

fat16lib

#7
Sep 14, 2014, 07:04 pm Last Edit: Sep 14, 2014, 07:14 pm by fat16lib Reason: 1
Quote
If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.


Yes, your app has very simple filename usage. 8.3 file names have been good enough for a long time.  I think CP/M was the first OS with 8.3 filenames in 1973-1974.  The first FAT long filenames  were in Windows NT 3.5 in 1994.

CP/M 8.3 was an improvement over 6.3 in DEC TOPS-10,  RSX-11/M and early VAX VMS.  Valid 6.3 characters were 0-9 A-Z $ and %. filenames were encoded in RADIX-50, which packed those nine characters into only three 16-bit words (six bytes) http://en.wikipedia.org/wiki/DEC_Radix-50.   I thought 6.3 was good enough with DEC machines.

I am looking for examples where LFN support is needed or desired and the reason.

Edit: Here are the ugly details of encoding DEC filenames.
Quote

Encoding

All Radix-50 strings must be a multiple of three characters long and should be padded by spaces at the end if necessary. To encode the character sequence "xyz" into the 16-bit word e the following formula should be used:

   e = (((x × 40) + y) × 40) + z

CatweazleNZ

#8
Sep 16, 2014, 09:23 am Last Edit: Sep 16, 2014, 11:00 am by CatweazleNZ Reason: 1

Hi

If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.

Cheers

Catweazle NZ
My point is that the application environment for Arduino applications would not normally need long filenames.

I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser (or PC file download/processing). I would not expect an Arduino to be processing a lot of business software files (Excel, Word, Powerpoint, etc) which do generally benefit from long filenames to assist user file browsing at a later date.

The fact that data cannot be readily exchanged between an Arduino's SD card and a PC (without the hassle of shutting down the Arduino and removing the SD card) suggests to me that the data on an SD card is very likely to be restricted to the Arduino card's function and purpose.

Therefore (well it is my view anyway) a good directory/folder design and file naming convention according to the needs of the Arduino host application is easy enough to achieve with 8.3 filenames. (Remember a directory structure is a good substitute for long filenames anyway. My application filename /BACKUPS/2014/09/16/09161900.TXT/ obviously embeds hour, day, month & year and is a backup file with text content.)

That is not to say however that you should not try to implement long filename support - just for the fun of facing the challenge and scaling the mountain. And there will be a few grateful end-users who will make some use of long filenames on their Arduino application SD cards.

You also mentioned added RAM requirements. That might be your deciding factor - Arduino applications often hit the boundaries of available RAM (which is stolen so quickly by library objects associated with ethernet, SD card and other functionality - not to mention arrays, Strings and other application data) that the availability of a long filename SD card library with even greater RAM requirements may not be embraced by many who can get by with 8.3 filenames.

Cheers

Catweazle NZ

fat16lib

Quote
I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser


This is not true. You can not guess how diverse the apps are that use SD cards on Arduino.  Over 50,000 users have downloaded SdFat and I often hear from users with requests for new features.  SD.h is a wrapper for a simple early version of SdFat.  SdFat has evolved a great deal on the basis of these user requests.

I looked at you app and it is nice but it makes very simple use of files so I agree you don't need more.  I got your message you don't need LFN!

I have helped a number of users with minimal access to long file names and some users need more.   I want to hear from more users that need LFN so I can decide what to implement.

I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems.  No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.

SdFat is used in very sophisticated apps on Teensy 3.1 and Due.  These boards have lots of memory and developers want more features.

The most popular embedded File System, FatFs, provides full Long File Name Support in ANSI/OEM code pages or Unicode.  This includes Japanese Shift-JIS, Simplified Chinese, Korean EUC-KR, and Traditional Chinese.  I plan a far simpler implementation.

pico


I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems.  No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.


This sounds like the right way to go.


SdFat is used in very sophisticated apps on Teensy 3.1 and Due.  These boards have lots of memory and developers want more features.


I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.

So l for one would appreciate lfn support in SdFat. :-)
WiFi shields/Yun too expensive? Embeddedcoolness.com is now selling the RFXduino nRF24L01+ <-> TCP/IP Linux gateway: Simpler, more affordable, and even more powerful wireless Internet connectivity for *all* your Arduino projects! (nRF24L01+ shield and dev board kits available too.)

wildbill

I currently have no need for long filenames, but there are clearly some circumstances where it would be painful to rename everything on an SD card to 8.3. Given your proposed config option to make it optional, sounds like a worthy addition.

JChristensen

I haven't felt a need for LFN. If it adds significantly to processing overhead or to the code footprint, I'd prefer it be an option, I like SdFat a lot just like it is!

neil111

Quote
I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.

So l for one would appreciate lfn support in SdFat. :-)

I would ABSOLUTELY like to see LFNs available.  The suggestion to support 7 bit rather than 8 bit sounds like a perfect compromise.  Arduino applications need to be able to access files that originated outside of the Arduino environment (Mac, Windows, Linux), whether over Ethernet, BT, or from SD cards.

My own application needs to read files from SD cards that were created on a PC or Mac, so long file names is essential.

I will then need to display the filename on the TFT.  Of course handling excessively LFNs would require additional processing for the TFT display, but that's fine as far as I'm concerned.

Any thoughts on when a beta implementation might be available?
Thanks!
Neil

fat16lib

Almost all requests have been for opening existing files so I think I will implement that and wait for more interest in creating lfn files.

Opening existing lfn files is not too difficult.  I just need to decide what the API should be.  In addition to open for a lfn, I need a function for finding all files and returning both short and long names.

Here is output from the test program below that lists all files in a directory.  The file name in the first column is the short name.

Quote
THELIN~1.PDF  The Linux Programming Interface.pdf
SCOTT-~1.MP3  scott-joplin-peacherine-rag.mp3
DAISY-~1.MP3  daisy-bell.mp3
ITS-A-~1.MP3  its-a-long-long-way-to-tipperary.mp3
SHORT.TXT
lower.txt 8.3 lowercase bits: 18
MIXED.TXT  Mixed.TXT
Done
Code: [Select]

#include <SdFat.h>
SdFat sd;
const uint8_t SD_CS_PIN = SS;
//------------------------------------------------------------------------------
// Implements directory byte 12 for lowercase bits with 8.3 names.
// bit 0X10 means lowercase extension and bit 0X08 lowercase basename
void listLfn(SdBaseFile* dirFile) {
  uint8_t offset[] = {1, 3, 5, 7, 9, 14, 16, 18, 20, 22, 24, 28, 30};
  char name[13];
  char lfn[131];
  bool haveLong = false;
  dir_t dir;
  uint8_t i;
  uint8_t lfnIn = 130;
  uint8_t ndir;
  uint8_t sum;
  uint8_t test;

  dirFile->rewind();
  while (dirFile->read(&dir, 32) == 32) {
    if (DIR_IS_LONG_NAME(&dir)) {
      if (!haveLong) {
        if ((dir.name[0] & 0XE0) != 0X40) continue;
        ndir = dir.name[0] & 0X1F;
        test = dir.creationTimeTenths;
        haveLong = true;
        lfnIn = 130;
        lfn[lfnIn] = 0;
      } else if (dir.name[0] != --ndir || test != dir.creationTimeTenths) {
        haveLong = false;
        continue;
      }
      char *p = (char*)&dir;
      if (lfnIn > 0) {
        lfnIn -= 13;
        for (i = 0; i < 13; i++) {
          lfn[lfnIn + i] = p[offset[i]];
        }
      }
    } else if (DIR_IS_FILE_OR_SUBDIR(&dir)
      && dir.name[0] != DIR_NAME_DELETED
      && dir.name[0] != DIR_NAME_FREE) {
      if (haveLong) {
        for (sum = i = 0; i < 11; i++) {
           sum = (((sum & 1) << 7) | ((sum & 0xfe) >> 1)) + dir.name[i];
        }
        if (sum != test || ndir != 1) haveLong = false;
      }
      SdFile::dirName(dir, name);
      if (dir.reservedNT) {
        bool dot = false;
        for (char *p = name; *p; p++) {
          if (*p == '.') {
            dot = true;
            continue;
          }
          if (dot && (dir.reservedNT & 0X8)
            || !dot && (dir.reservedNT & 0X10)) {
              *p = tolower(*p);
            }
        }
      }
      Serial.print(name);
      if (haveLong) {
        Serial.print("  ");
        Serial.print(lfn + lfnIn);
      }
      if (dir.reservedNT) {
        Serial.print(" 8.3 lowercase bits: ");
        Serial.print(dir.reservedNT, HEX);
      }
      Serial.println();
      haveLong = false;
    } else {
      if (dir.name[0] == DIR_NAME_FREE) return;
      haveLong = false;
    }
  }
}
//------------------------------------------------------------------------------
void setup() {
  Serial.begin(9600);
  if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
  // list files in root directory
  listLfn(sd.vwd());
  Serial.println("Done");
}
void loop() {}

Go Up