I think that generally Arduino applications are limited to writing log files, reading files in limited ways and perhaps sending them down an ethernet connection (or other wire) for display within a browser
This is not true. You can not guess how diverse the apps are that use SD cards on Arduino. Over 50,000 users have downloaded SdFat and I often hear from users with requests for new features. SD.h is a wrapper for a simple early version of SdFat. SdFat has evolved a great deal on the basis of these user requests.
I looked at you app and it is nice but it makes very simple use of files so I agree you don't need more. I got your message you don't need LFN!
I have helped a number of users with minimal access to long file names and some users need more. I want to hear from more users that need LFN so I can decide what to implement.
I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems. No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.
SdFat is used in very sophisticated apps on Teensy 3.1 and Due. These boards have lots of memory and developers want more features.
The most popular embedded File System, FatFs, provides full Long File Name Support in ANSI/OEM code pages or Unicode. This includes Japanese Shift-JIS, Simplified Chinese, Korean EUC-KR, and Traditional Chinese. I plan a far simpler implementation.
fat16lib:
I plan to add Long File Names in the same way it is implemented in other popular FAT libraries for embedded systems. No extra flash will be used unless Long File Names are enabled and extra RAM will be allocated as temp storage on the stack so no extra RAM will be required unless you call Long File Name member functions.
This sounds like the right way to go.
fat16lib:
SdFat is used in very sophisticated apps on Teensy 3.1 and Due. These boards have lots of memory and developers want more features.
I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.
So l for one would appreciate lfn support in SdFat.
I currently have no need for long filenames, but there are clearly some circumstances where it would be painful to rename everything on an SD card to 8.3. Given your proposed config option to make it optional, sounds like a worthy addition.
I haven't felt a need for LFN. If it adds significantly to processing overhead or to the code footprint, I'd prefer it be an option, I like SdFat a lot just like it is!
I've actually got a Teensy 3 project that would benefit from lfn support, to be able tp read .mp3 files from a sd card and display the long name on a tft display along with other info.
So l for one would appreciate lfn support in SdFat.
I would ABSOLUTELY like to see LFNs available. The suggestion to support 7 bit rather than 8 bit sounds like a perfect compromise. Arduino applications need to be able to access files that originated outside of the Arduino environment (Mac, Windows, Linux), whether over Ethernet, BT, or from SD cards.
My own application needs to read files from SD cards that were created on a PC or Mac, so long file names is essential.
I will then need to display the filename on the TFT. Of course handling excessively LFNs would require additional processing for the TFT display, but that's fine as far as I'm concerned.
Any thoughts on when a beta implementation might be available?
Thanks!
Neil
Almost all requests have been for opening existing files so I think I will implement that and wait for more interest in creating lfn files.
Opening existing lfn files is not too difficult. I just need to decide what the API should be. In addition to open for a lfn, I need a function for finding all files and returning both short and long names.
Here is output from the test program below that lists all files in a directory. The file name in the first column is the short name.
THELIN~1.PDF The Linux Programming Interface.pdf
SCOTT-~1.MP3 scott-joplin-peacherine-rag.mp3
DAISY-~1.MP3 daisy-bell.mp3
ITS-A-~1.MP3 its-a-long-long-way-to-tipperary.mp3
SHORT.TXT
lower.txt 8.3 lowercase bits: 18
MIXED.TXT Mixed.TXT
Done
#include <SdFat.h>
SdFat sd;
const uint8_t SD_CS_PIN = SS;
//------------------------------------------------------------------------------
// Implements directory byte 12 for lowercase bits with 8.3 names.
// bit 0X10 means lowercase extension and bit 0X08 lowercase basename
void listLfn(SdBaseFile* dirFile) {
uint8_t offset[] = {1, 3, 5, 7, 9, 14, 16, 18, 20, 22, 24, 28, 30};
char name[13];
char lfn[131];
bool haveLong = false;
dir_t dir;
uint8_t i;
uint8_t lfnIn = 130;
uint8_t ndir;
uint8_t sum;
uint8_t test;
dirFile->rewind();
while (dirFile->read(&dir, 32) == 32) {
if (DIR_IS_LONG_NAME(&dir)) {
if (!haveLong) {
if ((dir.name[0] & 0XE0) != 0X40) continue;
ndir = dir.name[0] & 0X1F;
test = dir.creationTimeTenths;
haveLong = true;
lfnIn = 130;
lfn[lfnIn] = 0;
} else if (dir.name[0] != --ndir || test != dir.creationTimeTenths) {
haveLong = false;
continue;
}
char *p = (char*)&dir;
if (lfnIn > 0) {
lfnIn -= 13;
for (i = 0; i < 13; i++) {
lfn[lfnIn + i] = p[offset[i]];
}
}
} else if (DIR_IS_FILE_OR_SUBDIR(&dir)
&& dir.name[0] != DIR_NAME_DELETED
&& dir.name[0] != DIR_NAME_FREE) {
if (haveLong) {
for (sum = i = 0; i < 11; i++) {
sum = (((sum & 1) << 7) | ((sum & 0xfe) >> 1)) + dir.name[i];
}
if (sum != test || ndir != 1) haveLong = false;
}
SdFile::dirName(dir, name);
if (dir.reservedNT) {
bool dot = false;
for (char *p = name; *p; p++) {
if (*p == '.') {
dot = true;
continue;
}
if (dot && (dir.reservedNT & 0X8)
|| !dot && (dir.reservedNT & 0X10)) {
*p = tolower(*p);
}
}
}
Serial.print(name);
if (haveLong) {
Serial.print(" ");
Serial.print(lfn + lfnIn);
}
if (dir.reservedNT) {
Serial.print(" 8.3 lowercase bits: ");
Serial.print(dir.reservedNT, HEX);
}
Serial.println();
haveLong = false;
} else {
if (dir.name[0] == DIR_NAME_FREE) return;
haveLong = false;
}
}
}
//------------------------------------------------------------------------------
void setup() {
Serial.begin(9600);
if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
// list files in root directory
listLfn(sd.vwd());
Serial.println("Done");
}
void loop() {}
In a second step, as I do not want to remove the SD card each time I add some music files (for copying them in my PC), I decide to include a FTP server in my mp3 reader. I had published some versions of my FTP server (Ftp Server on Arduino - Networking, Protocols, and Devices - Arduino Forum). For now, it only works with short name files.
As I do not know how to create long name files, I wrote a version of my program that use a table stored in a file in each directory. That table allow me to convert long names to short names (I have not yet published it). It works, but it is not the perfect solution.
So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.
So I would be VERY interested in a way to write the files with their long names!
And, if you need it, I am ok to test some beta version of your code.
If I add file creation for long file names to SdFat, it must work on Uno with minimal use of memory. Looks like you are using Due so that is not my first choice for testing.
There are lots of LFN implementations that work on ARM with more RAM. I am starting to develop very memory efficient algorithms suitable for LFN creation on Uno. This is a low priority effort since few users need file creation.
I will first finish open for existing LFN files since that is needed to test for an existing file before creating a new file. I think I can do this with no internal buffering for strings. This is important since an LFN can be 255 characters long.
My next priority will be an openNext() that returns the file name to a user specified buffer. This will allow the user to limit the length of the returned name and save RAM.
There are lots of LFN implementations that work on ARM
So I search on the web. I don't found anything except ChibiOS/RT. And, good news, I discover you worked with this OS on Arduino.
So I have two questions:
1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?
2- Can you tell me what other long file names implementations work on Due?
Perhaps my questions are far from the main subject of your post and it would be better to create a new one?
Jean-Michel
1- About ChibiOS/RT: I see there is support for long file names ( os/fs/fatfs directory ) through a library written by Chan (http://elm-chan.org). But should I write the low-level functions specific of Due to access the SD memory?
I can't tell you if you should write functions for Due. I use STM32 boards with ChibiOS/RT and FatFS is well supported so I just use Giovanni's port.
2- Can you tell me what other long file names implementations work on Due?
I only use Due for minimal testing of SdFat. I don't use Due for projects so I don't know what is available. I think Atmel has FatFS ports for SAM3X. I think there may also be a FreeRTOS port.
I will soon release a version of SdFat with some LFN support. I plan to extend this to file creation and deletion in the near future.
Here is an example program that lists the long and short file names in a directory and prints the contents of user selected files. This should be enough for many read-only applications.
// Example use of openNextLFN and open by index.
#include <SdFat.h>
#include <SdFatUtil.h>
const uint8_t SD_CS_PIN = SS;
SdFat sd;
SdBaseFile file;
uint16_t index[10];
uint16_t n = 0;
//------------------------------------------------------------------------------
void setup() {
char name[50];
dir_t dir;
Serial.begin(9600);
Serial.println(F("\r\nTest files: SdFat/examples/LongFileName/testFiles"));
if (!sd.begin(SD_CS_PIN)) sd.initErrorHalt();
Serial.print(F("Free RAM: "));
Serial.println(FreeRam());
Serial.println();
// list files in root directory
sd.vwd()->rewind();
while (file.openNextLFN(sd.vwd(), name, sizeof(name), O_READ) > 0) {
if (!file.dirEntry(&dir)) sd.errorHalt(F("dirEntry"));
file.close();
// Skip directories and hidden files.
if (DIR_IS_HIDDEN(&dir) || DIR_IS_SUBDIR(&dir)) continue;
// Save index of file in directory.
index[n] = sd.vwd()->curPosition()/32 - 1;
Serial.print(n);
Serial.write(' ');
Serial.println(name);
if (++n == 10) break;
}
}
//------------------------------------------------------------------------------
void loop() {
int c;
while (Serial.read() > 0) {}
Serial.print(F("\r\nEnter File Number: "));
while ((c = Serial.read()) < 0) {};
if (!isdigit(c) || (c -= '0') >= n) {
Serial.println(F("Invald number"));
return;
}
Serial.println(c);
if (!file.open(sd.vwd(), index[c], O_READ)) {
sd.errorHalt(F("open"));
}
Serial.println();
char last;
while ((c = file.read()) > 0) Serial.write(last = (char)c);
// Print a new line if needed.
if (last != '\n') Serial.println();
file.close();
}
This is the size of the program.
Binary sketch size: 10,712 bytes (of a 32,256 byte maximum)
Here is output from the program. First the program lists the files in the directory. It then asks the user to select a file. The content of the file is then printed.
Free RAM: 1094
0 With Two.dots.txt
1 A long name can be 255 characters.txt
2 LFN,NAME.TXT
3 lower.txt
4 MIXCASE.txt
5 mixed.TXT
6 Not_8_3.txt
7 OK%83.TXT
8 STD_8_3.TXT
9 With Blank.txt
Enter File Number: 1
This is "A long name can be 255 characters.txt"
It is has a typical Long File Name.
Enter File Number: 2
LFN,NAME.TXT is not 8.3 since it has a comma.
Enter File Number: 5
mixed.TXT does not have a Long File Name.
Starting with NT, file names of this form,
have the basename and extension character case
encoded in two bits of the 8.3 directory entry.
NB here, at least as far as using 'duinos, etc. Lots of experience in DAS, DBMSs, some web, other stuff. This LFN post was prescient for me. Building a 'normal' data logging set of Teensy 3.1s (humidity, multiple temps, checking for water incursion, overflows, etc) - home app. Not there yet, but next level will be using PI, Beagle, or Galileo (collectors - have to 'play' first). Anyway, for data logging, I've 'always' used file names formed like:
_yyyy-mm-dd_hh:mm:ss. (obviously on the 'larger' machines).
An example might be: Bldg411_2014-06-03_23:28:42.txt. That way, I can group multiple sets of logs together, sorting them in asc or desc order. Always a form of text for content, sometimes XML, sometimes TSV, CSV (don't do that too much). Content tells me which collection source (intend to use right side MAC for T3), which measurement, UTC (IMHO, should ALWAYS store DAS data in UTC time coords), data value (in raw, or Eng-unit converted if that's all I can get). I use SD to store based on DAS req't time entropy so I don't lose it (hate to lose data, and to get from when file(s) are requested from collector). Collection based on
So re LFN, I'd like to create file(s). Intend to create new file during run once stage. As I said, using Teensy 3.1s. Started with Unos, then on to 2560s (which are fine), then to Teensy 3.1s. If exist (not expected), open with append, else create new file. Any timeframe re creation / open with intent to write / append?
If people want basic LFN support in SdFat now, they can take a look at the portions of SdFat that are included in the ErikZalm/Marlin project on GitHub. (For example, search for "longFilename" in this file.) LFN support was added to the code some time ago, and it's definitely something of a hack, but it works. It only supports 8-bit characters.
As a contributor to Marlin, I know we would rather have formal LFN support and include SdFat as an external instead of needing to include a modified copy of SdFat in Marlin's sketch folder.
I look forward to trying out the beta and seeing how easily it can be integrated into the Marlin codebase. Last time I tried including SdFat as an external my Arduino software wouldn't see it (...grumble...) so for the time-being I'm just dropping the newer SdFat into the sketch.
Wish us luck! We're doing some housecleaning on Marlin to bring things up to speed and clear out the backlog.
I just implemented the SDFat library on an Arduino Ethernet board with no issues.
We wanted a way to save service files inside our TCP connected equipment and your library made it a breeze so thank you very much.
I did this under IDE 1.0.6 and the SPI at full speed. I had no issues at all with your code.
I recently decided to try and build an music player using a hardware decoder IC (VS1053) as a slave and Arduino as the user interface and file system...
I was hoping to copy my files directly from the computer to the SD card or USB with file names in tact... For my part, I tend to restrict file names to 32 characters even on the PC, so limiting long file names is not an issue.
The final project intends to use a decent TFT display with either touch sensor or capacitive keys (not decided) and should for the most part provide a solution similar to my favourite DOS MP3 player... MPXPlay.
While the same could be accomplished with 8.3, it complicates the process, so LFN support would help me greatly... not to mention making the project more agreeable and modern...