Functions to wrap FatFs on Arduino

Using Arduino Storage
1

A new version is available - See reply #18
I wrote some simple functions to wrap FatFs module on Arduino.

FatFs is a generic FAT file system module for small embedded systems. It is developed by ChaN
See FatFs - Generic FAT Filesystem Module

This permits full access (list, open, read, write, rename,...) with long name to files and directories stored on SD cards

I tested it on Arduino Due and Arduino Mega 2560 with IDE 1.5.8

It is based on version R0.10b of FatFs
I use functions included in SD library (files Sd2Card.h and Sd2Card.cpp) to access the card reader through SPI,
so it is necessary to include SD.h in the sketch

I create a repository on GitHub
FatFs files are included in the repository because of I had to slightly modify some of those files.

File FatFsTest.ino includes wrapping functions and the demo of most basic functions of FatFs

Any comment and test would be appreciated

2

Thanks for the FatFS example. I was about to do this so I could compare the new LFN version of SdFat I am developing with FatFS. I wanted to see how much RAM FatFS uses for LFN. I want Long File Names to be practical on Uno.

You use lots of RAM for string literals and your program tests lots of FatFS features but it still appears I am way ahead of FatFS for RAM use.

Your test program on a Mega:

Sketch uses 27,694 bytes (10%) of program storage space. Maximum is 253,952 bytes.
Global variables use 4,219 bytes (51%) of dynamic memory, leaving 3,973 bytes for local variables. Maximum is 8,192 bytes.

This little test program creates a directory and a file with Long File Names. It then writes a line of text to the file.

#include <SPI.h> 
#include <SdFat.h>

SdFat sd;
SdFile dir;
SdFile file;

void setup() {
  Serial.begin(9600);

  if (!sd.begin()) {
    Serial.println("begin failed");
    return;
  }
 
  dir.mkdir(sd.vwd(), "A test directory");

  file.open(&dir, "A SdFat LFN.txt", O_RDWR | O_CREAT);

  file.println("Hello");

  file.close();
  Serial.println("Done");
}
void loop() {}

It uses this much memory on a Mega.

Sketch uses 12,972 bytes (5%) of program storage space. Maximum is 253,952 bytes.
Global variables use 935 bytes (11%) of dynamic memory, leaving 7,257 bytes for local variables. Maximum is 8,192 bytes.

Here is a list of the SD on a PC.

H:>ls -Rl
.:
total 0
drwxr-xr-x 1 bill None 0 Jan 1 2000 A test directory

./A test directory:
total 16
-rw-r--r-- 1 bill None 7 Jan 1 2000 A SdFat LFN.txt

The new SdFat is based on a new generic FAT12/FAT16/FAT32 core, like FatFS, so I may try USB mass storage next. Not likely that will fit a Uno.

3

Thank you for your interest.
No doubt SdFat is more efficient than FatFs and my wrapper, not only on memory usage but also on velocity.

I have a FTP server running on Arduino Due using your SdFat library with short names.
I modify this server to use FatFs, so I can store/retrieve files and directories with long names. But the byte rate has been reduced by about two. I guess it comes from the way in which the SD library access the card reader, less efficient than SdFat.

So it will be interesting to be able to use SdFat again for this project.

4

I wrote a library to use FatFs on Arduino Due or Mega2560 using classes.
It is on GitHub: GitHub - gallegojm/Arduino-FatFs: Functions to wrap FatFs module developed by ChaN on Arduino platform
It include three classes:

  • FatFsClass
  • DirFs
  • FileFs
    I tried as much as possible to use the same function names and the same calling conventions as those of SdFat library.
    An example FatFsDemo show the use of most functions.
5

I did a benchmark of the new SdFat vs this port of FatFS on a Due. I did a quick and dirty mod of the SdFat bench example. I attached the program.

At first I got data verify errors on tests of FatFs. You need to modify sd_disk_read() and sd_disk_write() something like this. note the n*512 additions.

extern "C" int sd_disk_read( uint8_t * buff, uint32_t sector, uint32_t count )
{
  uint8_t * b = buff;
  
  for( int n = 0; n < count; n ++ )
    if( card.readBlock( sector + n, b + n*512) == 0 )  // <<----- add n*512
      return 1;
  return 0;
}

extern "C" int sd_disk_write( uint8_t * buff, uint32_t sector, uint32_t count )
{
  uint8_t * b = buff;
  
  for( int n = 0; n < count; n ++ )
    if( card.writeBlock( sector + n, b + n*512 ) == 0 )  // <<----- add n*512
      return 1;
  return 0;
}

Here are the results for FatFs. The test writes a 5 MB file with writes of the size in the first column. The write rate in KB/sec is in the second column. The test then reads the file with reads the same size. The read rate in KB/sec is in the third column.

FatFs benchmark

size,write KB/sec,read KB/sec
100,171.39,242.25
512,173.14,254.33
1024,176.53,252.51
2048,176.65,252.79
32768,176.78,253.02

Here are results from the same test with SdFat.

SdFat benchmark

size,write KB/sec,read KB/sec
100,443.77,1477.54
512,443.67,1646.26
1024,837.24,1796.97
2048,1439.02,2631.14
32768,3667.70,4455.04

FatFS would be much faster with a better low level SD read/write driver but the Arduino Due SD.h library uses SPI.h at very slow speeds. I use a custom SPI with DMA.

The most important difference for real programs is for smaller, 100 byte, reads. FatFs does 242 KB/sec while SdFat does 1477 KB/sec. Few programs read/write in 32KB chunks but I did optimize SdFat for large transfers so it is fast.

FatFsBench.ino (4.44 KB)

6

Hi fat16lib
I appreciate your help and suggestions

You need to modify sd_disk_read() and sd_disk_write()

That is true. I never used transfer buffers greater than 512 bytes, so I do not detect my error. This is fixed in the repository.

An application for which I need long name files is a FTP server. I made some speed measurements with both libraries and varying the size of the read/write buffer. You can see them there:Ftp Server on Arduino - #45 by gallegojm - Networking, Protocols, and Devices - Arduino Forum

7

What type SD card are you using. There is a large difference for a given application. Expensive or higher class cards do not always perform best for a given app on Arduino.

SD cards are not primarily designed for the Arduino mode of use so you need to experiment for maximum performance.

If networking limits the total bandwidth to about 200 KB/sec, it may not matter, even the SPI library doesn't matter at low total speed.

SD cards have a curious performance profile. Two cards can have exactly the same timing up to some total bandwidth say 200 KB/sec. If you push these cards harder, one may start to go busy at times. This means that for low performance applications you can't see a difference between cards and there is no point in paying for more performance.

The same carries over to SD libraries. For most people there is no real gain in using the newer SdFat over SD.h with its older SdFat.

8

I put the latest SdFat SD driver in Your FatFS port. Here are the benchmark results.

size,write KB/sec,read KB/sec
100,332.42,1109.14
512,351.74,1107.84
1024,834.17,1786.69
2048,1432.84,2602.38
32768,3640.89,4423.39

SdFat and FatFS are identical for large transfers since the SD driver dominates. SdFat has a special optimization for small transfers so it is a bit faster where the FAT implementation matters.

For 10 byte writes, SdFat does 445 KB/sec and FatFS does 265 KB/sec using the same SD card and SD driver.

9

What type SD card are you using

For testing, I use an old 2GB 'SanDisk Premier' card.
After reading your message, I repeat some testing with a 'no name' card of 32GB than I reformat with SDFormatter.
I was surprised with the speed increment. Next I play with the size of the read/write buffer used for transferring the data over ethernet (my first choice of 512 and 2048 bytes was somewhat arbitrary) and got best results with a 1kB size. Speed rose to 246 kB/s writing some files to the card (STOR command).

I put the latest SdFat SD driver in Your FatFS port

This is in my 'to do list'. Of course I am very interested and I had studied your different implementations for driving the SPI bus. (Can you tell me how to do it?)
But for the moment I have the following problem: My Ftp Server hangs when I run it with SdFat setting SD_SPI_CONFIGURATION to 0 (for using your fast implementation). I guess this is a conflict with the SPI driver for the Ethernet circuit.
So I have to set SD_SPI_CONFIGURATION to 1. This is with this setting than I get the results I posted in the topic on FtpServer (I forgot to notify that point)

I have a particular project (I refer to it in the first message of the topic on FtpServer) where the Arduino Due reads a file to feed a mp3 decoder and at the same time can transfers files through the ftp server. So the Due access 'simultaneously' three devices through the SPI bus and I do not know if it is possible to use your fast implementation.

In conclusion, I saw it is very difficult to optimize every parameters and obtain the best compromise varies from one application to another.

10

My Ftp Server hangs when I run it with SdFat setting SD_SPI_CONFIGURATION to 0 (for using your fast implementation). I guess this is a conflict with the SPI driver for the Ethernet circuit.

Every SPI device library must setup the SPI bus for each access. SdFat initializes the SPI bus before each access and uses DMA on Due. This is the only way to get fast SD transfers in the 4 MB/sec range.

The Ethernet library does not appear to restore settings yet.

The Arduino company has recognized this problem and is introducing an elaborate scheme to solve the settings problem and provide a way to share the SPI bus between interrupt and non-interrupt code.

I don't like the protocol for sharing the SPI bus between interrupt and non-interrupt code since it is very hardware dependent and better solutions were developed in the embedded world over 30 years ago. I added the option to enable "SPI transactions" so users can try to use the SPI bus in interrupt routines.

Too bad the Due doesn't have multiple SPI pins, almost all Cortex M chips now have multiple SPI controllers. Some STM32 chips have as many as five SPI controllers.

You can look at the new SPI driver code in this folder.

arduino-1.5.8\hardware\arduino\sam\libraries\SPI

I expect the Ethernet library will be updated but who knows about other third party libraries.

Update: I looked at the Ethernet library and it appears to be fixed in 1.5.8. Maybe the new SPI library has a problem coping with my use of DMA. I am not alone, other libraries use DMA SPI.

11

In conclusion, I saw it is very difficult to optimize every parameters and obtain the best compromise varies from one application to another.

This is true. You are not likely to test important features when you are developing an app. I keep improving parts of SdFat when users find performance problems. For example, the algorithm to find a free cluster when the SD has fragments.

You could might get a bit more performance by increasing the SPI speed for the Ethernet board from 4 MHz to 8 MHz.

You will never get great Ethernet performance from the W5100. It is a great part but the slow SPI kills performance.

You got me interested again in network performance for small embedded systems. I am starting development with this board. The cost is less than a Due plus R3 Ethernet shield. I bought it for $52.75 here.

The STM32F407 has a built-in MAC connected to the Ethernet PHY with a 2-bit wide 50 MHz link. I am using ChibiOS/RT as the RTOS and the lwip network stack.

The SD on this board is connected to the chip with a 4-bit SDIO bus so it can do file writes at over 10 MB/sec with my FAT library.

I have a few demos going, including a little web server that is really fast. ftp would be interesting.

12

You will never get great Ethernet performance from the W5100

Yes, that is why I use an Wiz820io that include a fast W5200

I am starting development with this board

The STM32E407 looks very interesting! I read the specification and I really consider to buy one. Which Jtag interface did you use? (Is it mandatory ? I have none for the moment) And which software did you use to develop ?
I make a fast research on the web and see this tuto : http://www.angstromsandalgorithms.com/free-eclipse-arm-gcc-openocd-toolchain-for-windows-part-1-introduction/

13

Yes, that is why I use an Wiz820io that include a fast W5200

It is still a dog with a slow SPI connection. A true MAC - PHY connection with DMA is 100 Mbit/sec.

The STM32E407 looks very interesting! I read the specification and I really consider to buy one. Which Jtag interface did you use? (Is it mandatory ? I have none for the moment)

I use ST-Link for STM32 ARM chips. You can program STM32 chips with a DFU, Device Firmware Update, loader but I find ST-Link much better. You can use ST-Link to watch memory or run a true debugger. Memory watch is nice since the program is not aware and executes at full speed.

See this for an example of the type display STMStudio provides using symbols from the elf file.

There are other boards that are much nicer to program. These boards implement the mbed drag and drop loader. You see your board as a flash drive and just drop the bin file into the drive. I just bought two of these which have the same chip as the STM32-E407. They are very new so may be a risk. The STM32-407E is a super solid board by Olimex and has had several revisions.

And which software did you use to develop ?

I use the free tool chain by ARM.

I use ChibiOS/RT with the version of lwIP that has been patched for the RTOS.

ChibiOS/RT can do a context switch in a half microsecond on the 407 boards so you get true commercial level RTOS performance for your tasks.

There are lots of network projects based on lwIP so you can start with great examples.

The down side is that users have lots of problems optimizing lwIP, there are something like 200 options in the lwIP opt.h file. The opt.h file is 2100 lines long. You need good instrumentation to understand what is happening.

The WIZnet products are great for most users.

Edit: Here are some lwIP tests on NXP ARM achieving about 88 Mbits/sec.

14

I create a library FatLib that allow to select the library (FatFs or SdFat) for accessing the SD card changing lonely the value of an identifier ( #define FAT_SYST at the beginning of the file FatLib/FatLib.h )

Of course, their are some compromises in the name of the classes, functions and parameters, but I try to take the best of the two worlds...
You have to update FatFs.

Have a look at the example ( FatLib/examples/FatLibDemo/FatLibDemo.ino ) to see how to use it.
I rewrote too my FtpServer accordingly to these changes.

The code is at Github

15

As suggest fat16lib ("I put the latest SdFat SD driver in Your FatFS port"), I replace the SPI driver of library SD with the drivers of SdFat.
I made some tests (Arduino Due, Ide 1.5.6-r2 and a fresh formated old 2GB micro SD card)
I adapt the benchmark written by fat16lib (FatFs/examples/FatFsBench).
I run it with the last version of FatFs that use the SPi driver of library SD
Then I run it with the new version of FatFs that include SdSpiCard.h. I do it with SD_SPI_CONFIGURATION set to 1 and then to 0 (in file SdFatConfig.h)
I repeat this with SdFat/examples/bench
I made this 5 tests with a read/write buffer of 100 and 512 bytes
Results are in the table below.

Speeds of FatFs with new drivers are higher than before although still lower than the ones obtained with SdFat.
I obtain higher speeds in writing/reading although still lower than the one obtained with SdFat.
Ram usage is reduced too, but remains much higher than with SdFat.
Sketch size, which was already higher than the equivalent in SdFat, is incremented by about 8 kbytes
So, why to use FatFs?
I see a lonely reason: if you need to manage file's name with extended characters (see ffconf.h for the different options for selecting the code page and the character encoding)
The code is at Github

benchmark.jpg

16

what provides SdSpiCard.h ?

inFatFs/FatFs.h:
#include <SdSpiCard.h>

17

The current version of SdSpiCard is in this folder.

The version with FatFs may be modified and/or older.

18

It was difficult to adapt the software of my projects in the last IDE 1.5.8 due to changes in the SPI drivers.
I decided to focus only on the Arduino Due for this upgrade.
It includes:

  • the new module R0.11 of FatFs by ChaN

  • new functions to read and modify the last time files were modified
    For the modified files have their time modified automatically, you need to provide the time information through the function get_fattime() in file FatFs.cpp. This is dependent of your hardware.

  • SPI driver for SD card on Due
    this driver is an adaptation from the SPI driver of library SdFat by William Greiman. So it is no more necessary to include SdSpiCard.h in your sketch

This upgrade is at GitHub

19

I upgrade the library FatFs (download on Github)

  • use version R0.12c of FatFs by ChaN
  • use low level routines from SD library to access the card with Esp8266 chip
  • use low level routines from SdFat library with other chips, so you have to install SdFat library in order to use FatFs

I upgrade the library FatLib (download on Github)
FatLib allow now to switch between SdFat, FatFs and SpiFfs by modifying only one line in your project.
See Readme.md for a list of classes and functions you must use instead of original functions of SdFat, SD, FatFs or SpiFfs
(note: SpiFfs is the library that permits to simulate a file system with part of the flash memory of an Esp8266)