I would appreciate comments on the need for LFN, Long File Names, in SdFat.
I have not implemented LFN because of the complexity, amount of flash, and RAM required. I am now looking at making LFN a configuration option so the default size of SdFat would not increase.
LFN uses UTF-16 characters but I only plan on supporting an 8-bit character set.
LFN is slow and uses lots of memory.
The long file name system allows a maximum length of 255 UTF-16 characters, including spaces and non-alphanumeric characters. This is achieved by chaining up to 20 directory entries of 13 2-byte unicode characters each.
Allowing each portion of a path name to be 255 characters long requires lots of memory. Some implementations of FAT for embedded systems limit the maximum length of a file/directory name. I plan to provide a configuration parameter so small memory Arduinos like Uno can use limited LFN.
LFN might make sense on the DUE which has more RAM and processing power.
For the UNO/MEGA scope I agree with Peter,
reading them would be the first need.
rename them to an 8.3 file name would be the second need
some questions came up?
are there other file formats for SD cards e.g. those from linux that could be applied to an SD that are faster in general?
Can an SDcard have partitions?
When you think about other file formats, you could do simple binary writes without file system and read and convert it in linux. It is not that hard in linux.
In the past, some USB memory sticks did not have a partition table, just a straight forward FAT16 file system. As far as I know, today most memory sticks/cards have a partition table with a single partition. So you you can have many partitions, just as with a harddisk.
Files created by Arduino : LFN not needed.
But I would like to open files created by a computer with a long name.
This sounds like a good first step. I already have a prototype working. The prototype supports 7-bit characters, the original ASCII ASCII - Wikipedia. Extending to 8-bit SBCS code pages or UTF-16 adds a huge amount of complexity.
The the extra characters in standard U.S. Microsoft code page 437 are now anarchic Code page 437 - Wikipedia.
are there other file formats for SD cards e.g. those from linux that could be applied to an SD that are faster in general?
Can an SDcard have partitions?
Long File Names do not change the speed of I/O. Just the time to open/create files due to the directory complexity.
Linux/Unix file systems are way too complex for Arduino and would not be faster on SD cards. The flash structure of SD cards is designed for FAT file systems.
The fastest way to access a file on an SD is to create a contiguous file and access it using multi-block raw SD reads and writes. SD cards are fast only if you use very large multi-block transfers since the flash pages are now some multiple of 16 KB.
As far as I know, today most memory sticks/cards have a partition table with a single partition. So you you can have many partitions, just as with a harddisk.
The base code in SdFat allows four partitions on an SD card or a single floppy style volume. The SD standard now only permits a single partition with a MBR so this feature is hidden.
If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.
If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.
Yes, your app has very simple filename usage. 8.3 file names have been good enough for a long time. I think CP/M was the first OS with 8.3 filenames in 1973-1974. The first FAT long filenames were in Windows NT 3.5 in 1994.
CP/M 8.3 was an improvement over 6.3 in DEC TOPS-10, RSX-11/M and early VAX VMS. Valid 6.3 characters were 0-9 A-Z $ and %. filenames were encoded in RADIX-50, which packed those nine characters into only three 16-bit words (six bytes) DEC RADIX 50 - Wikipedia. I thought 6.3 was good enough with DEC machines.
I am looking for examples where LFN support is needed or desired and the reason.
Edit: Here are the ugly details of encoding DEC filenames.
Encoding
All Radix-50 strings must be a multiple of three characters long and should be padded by spaces at the end if necessary. To encode the character sequence “xyz” into the 16-bit word e the following formula should be used:
If you browse the directories (folders) on my SD card for my application at http://www.2wg.co.nz you will see that my application does just fine with 8.3 filenames.
Cheers
Catweazle NZ
My point is that the application environment for Arduino applications would not normally need long filenames.
I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser (or PC file download/processing). I would not expect an Arduino to be processing a lot of business software files (Excel, Word, Powerpoint, etc) which do generally benefit from long filenames to assist user file browsing at a later date.
The fact that data cannot be readily exchanged between an Arduino's SD card and a PC (without the hassle of shutting down the Arduino and removing the SD card) suggests to me that the data on an SD card is very likely to be restricted to the Arduino card's function and purpose.
Therefore (well it is my view anyway) a good directory/folder design and file naming convention according to the needs of the Arduino host application is easy enough to achieve with 8.3 filenames. (Remember a directory structure is a good substitute for long filenames anyway. My application filename /BACKUPS/2014/09/16/09161900.TXT/ obviously embeds hour, day, month & year and is a backup file with text content.)
That is not to say however that you should not try to implement long filename support - just for the fun of facing the challenge and scaling the mountain. And there will be a few grateful end-users who will make some use of long filenames on their Arduino application SD cards.
You also mentioned added RAM requirements. That might be your deciding factor - Arduino applications often hit the boundaries of available RAM (which is stolen so quickly by library objects associated with ethernet, SD card and other functionality - not to mention arrays, Strings and other application data) that the availability of a long filename SD card library with even greater RAM requirements may not be embraced by many who can get by with 8.3 filenames.
I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser
This is not true. You can not guess how diverse the apps are that use SD cards on Arduino. Over 50,000 users have downloaded SdFat and I often hear from users with requests for new features. SD.h is a wrapper for a simple early version of SdFat. SdFat has evolved a great deal on the basis of these user requests.
I looked at you app and it is nice but it makes very simple use of files so I agree you don't need more. I got your message you don't need LFN!
I have helped a number of users with minimal access to long file names and some users need more. I want to hear from more users that need LFN so I can decide what to implement.
I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems. No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.
SdFat is used in very sophisticated apps on Teensy 3.1 and Due. These boards have lots of memory and developers want more features.
The most popular embedded File System, FatFs, provides full Long File Name Support in ANSI/OEM code pages or Unicode. This includes Japanese Shift-JIS, Simplified Chinese, Korean EUC-KR, and Traditional Chinese. I plan a far simpler implementation.
fat16lib:
I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems. No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.
This sounds like the right way to go.
fat16lib:
SdFat is used in very sophisticated apps on Teensy 3.1 and Due. These boards have lots of memory and developers want more features.
I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.
So l for one would appreciate lfn support in SdFat.
I currently have no need for long filenames, but there are clearly some circumstances where it would be painful to rename everything on an SD card to 8.3. Given your proposed config option to make it optional, sounds like a worthy addition.
I haven't felt a need for LFN. If it adds significantly to processing overhead or to the code footprint, I'd prefer it be an option, I like SdFat a lot just like it is!
I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.
So l for one would appreciate lfn support in SdFat.
I would ABSOLUTELY like to see LFNs available. The suggestion to support 7 bit rather than 8 bit sounds like a perfect compromise. Arduino applications need to be able to access files that originated outside of the Arduino environment (Mac, Windows, Linux), whether over Ethernet, BT, or from SD cards.
My own application needs to read files from SD cards that were created on a PC or Mac, so long file names is essential.
I will then need to display the filename on the TFT. Of course handling excessively LFNs would require additional processing for the TFT display, but that's fine as far as I'm concerned.
Any thoughts on when a beta implementation might be available?
Thanks!
Neil
Almost all requests have been for opening existing files so I think I will implement that and wait for more interest in creating lfn files.
Opening existing lfn files is not too difficult. I just need to decide what the API should be. In addition to open for a lfn, I need a function for finding all files and returning both short and long names.
Here is output from the test program below that lists all files in a directory. The file name in the first column is the short name.
THELIN~1.PDF The Linux Programming Interface.pdf
SCOTT-~1.MP3 scott-joplin-peacherine-rag.mp3
DAISY-~1.MP3 daisy-bell.mp3
ITS-A-~1.MP3 its-a-long-long-way-to-tipperary.mp3
SHORT.TXT
lower.txt 8.3 lowercase bits: 18
MIXED.TXT Mixed.TXT
Done
#include <SdFat.h>
SdFat sd;
const uint8_t SD_CS_PIN = SS;
//------------------------------------------------------------------------------
// Implements directory byte 12 for lowercase bits with 8.3 names.
// bit 0X10 means lowercase extension and bit 0X08 lowercase basename
void listLfn(SdBaseFile* dirFile) {
uint8_t offset[] = {1, 3, 5, 7, 9, 14, 16, 18, 20, 22, 24, 28, 30};
char name[13];
char lfn[131];
bool haveLong = false;
dir_t dir;
uint8_t i;
uint8_t lfnIn = 130;
uint8_t ndir;
uint8_t sum;
uint8_t test;
dirFile->rewind();
while (dirFile->read(&dir, 32) == 32) {
if (DIR_IS_LONG_NAME(&dir)) {
if (!haveLong) {
if ((dir.name[0] & 0XE0) != 0X40) continue;
ndir = dir.name[0] & 0X1F;
test = dir.creationTimeTenths;
haveLong = true;
lfnIn = 130;
lfn[lfnIn] = 0;
} else if (dir.name[0] != --ndir || test != dir.creationTimeTenths) {
haveLong = false;
continue;
}
char *p = (char*)&dir;
if (lfnIn > 0) {
lfnIn -= 13;
for (i = 0; i < 13; i++) {
lfn[lfnIn + i] = p[offset[i]];
}
}
} else if (DIR_IS_FILE_OR_SUBDIR(&dir)
&& dir.name[0] != DIR_NAME_DELETED
&& dir.name[0] != DIR_NAME_FREE) {
if (haveLong) {
for (sum = i = 0; i < 11; i++) {
sum = (((sum & 1) << 7) | ((sum & 0xfe) >> 1)) + dir.name[i];
}
if (sum != test || ndir != 1) haveLong = false;
}
SdFile::dirName(dir, name);
if (dir.reservedNT) {
bool dot = false;
for (char *p = name; *p; p++) {
if (*p == '.') {
dot = true;
continue;
}
if (dot && (dir.reservedNT & 0X8)
|| !dot && (dir.reservedNT & 0X10)) {
*p = tolower(*p);
}
}
}
Serial.print(name);
if (haveLong) {
Serial.print(" ");
Serial.print(lfn + lfnIn);
}
if (dir.reservedNT) {
Serial.print(" 8.3 lowercase bits: ");
Serial.print(dir.reservedNT, HEX);
}
Serial.println();
haveLong = false;
} else {
if (dir.name[0] == DIR_NAME_FREE) return;
haveLong = false;
}
}
}
//------------------------------------------------------------------------------
void setup() {
Serial.begin(9600);
if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
// list files in root directory
listLfn(sd.vwd());
Serial.println("Done");
}
void loop() {}
In a second step, as I do not want to remove the SD card each time I add some music files (for copying them in my PC), I decide to include a FTP server in my mp3 reader. I had published some versions of my FTP server (Ftp Server on Arduino - Networking, Protocols, and Devices - Arduino Forum). For now, it only works with short name files.
As I do not know how to create long name files, I wrote a version of my program that use a table stored in a file in each directory. That table allow me to convert long names to short names (I have not yet published it). It works, but it is not the perfect solution.
So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.
So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.
If I add file creation for long file names to SdFat, it must work on Uno with minimal use of memory. Looks like you are using Due so that is not my first choice for testing.
There are lots of LFN implementations that work on ARM with more RAM. I am starting to develop very memory efficient algorithms suitable for LFN creation on Uno. This is a low priority effort since few users need file creation.
I will first finish open for existing LFN files since that is needed to test for an existing file before creating a new file. I think I can do this with no internal buffering for strings. This is important since an LFN can be 255 characters long.
My next priority will be an openNext() that returns the file name to a user specified buffer. This will allow the user to limit the length of the returned name and save RAM.
There are lots of LFN implementations that work on ARM
So I search on the web. I don't found anything except ChibiOS/RT. And, good news, I discover you worked with this OS on Arduino.
So I have two questions:
1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?
2- Can you tell me what other long file names implementations work on Due?
Perhaps my questions are far from the main subject of your post and it would be better to create a new one?
Jean-Michel
1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?
I can't tell you if you should write functions for Due. I use STM32 boards with ChibiOS/RT and FatFS is well supported so I just use Giovanni's port.
2- Can you tell me what other long file names implementations work on Due?
I only use Due for minimal testing of SdFat. I don't use Due for projects so I don't know what is available. I think Atmel has FatFS ports for SAM3X. I think there may also be a FreeRTOS port.
I will soon release a version of SdFat with some LFN support. I plan to extend this to file creation and deletion in the near future.
Here is an example program that lists the long and short file names in a directory and prints the contents of user selected files. This should be enough for many read-only applications.
// Example use of openNextLFN and open by index.
#include <SdFat.h>
#include <SdFatUtil.h>
const uint8_t SD_CS_PIN = SS;
SdFat sd;
SdBaseFile file;
uint16_t index[10];
uint16_t n = 0;
//------------------------------------------------------------------------------
void setup() {
char name[50];
dir_t dir;
Serial.begin(9600);
Serial.println(F("\r\nTest files: SdFat/examples/LongFileName/testFiles"));
if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
Serial.print(F("Free RAM: "));
Serial.println(FreeRam());
Serial.println();
// list files in root directory
sd.vwd()->rewind();
while (file.openNextLFN(sd.vwd(), name, sizeof(name), O_READ) > 0) {
if (!file.dirEntry(&dir)) sd.errorHalt(F("dirEntry"));
file.close();
// Skip directories and hidden files.
if (DIR_IS_HIDDEN(&dir) || DIR_IS_SUBDIR(&dir)) continue;
// Save index of file in directory.
index[n] = sd.vwd()->curPosition()/32 - 1;
Serial.print(n);
Serial.write(' ');
Serial.println(name);
if (++n == 10) break;
}
}
//------------------------------------------------------------------------------
void loop() {
int c;
while (Serial.read() > 0) {}
Serial.print(F("\r\nEnter File Number: "));
while ((c = Serial.read()) < 0) {};
if (!isdigit(c) || (c -= '0') >= n) {
Serial.println(F("Invald number"));
return;
}
Serial.println(c);
if (!file.open(sd.vwd(), index[c], O_READ)) {
sd.errorHalt(F("open"));
}
Serial.println();
char last;
while ((c = file.read()) > 0) Serial.write(last = (char)c);
// Print a new line if needed.
if (last != '\n') Serial.println();
file.close();
}
This is the size of the program.
Binary sketch size: 10,712 bytes (of a 32,256 byte maximum)
Here is output from the program. First the program lists the files in the directory. It then asks the user to select a file. The content of the file is then printed.
Free RAM: 1094
0 With Two.dots.txt
1 A long name can be 255 characters.txt
2 LFN,NAME.TXT
3 lower.txt
4 MIXCASE.txt
5 mixed.TXT
6 Not_8_3.txt
7 OK%83.TXT
8 STD_8_3.TXT
9 With Blank.txt
Enter File Number: 1
This is "A long name can be 255 characters.txt"
It is has a typical Long File Name.
Enter File Number: 2
LFN,NAME.TXT is not 8.3 since it has a comma.
Enter File Number: 5
mixed.TXT does not have a Long File Name.
Starting with NT, file names of this form,
have the basename and extension character case
encoded in two bits of the 8.3 directory entry.
