Go Down

Topic: Don't Format SD cards with OS utilities! (Read 30921 times) previous topic - next topic

fat16lib

Always format SD cards using SD Formatter https://www.sdcard.org/downloads/formatter_4/.

NEVER format SD card with OS utilities!

Here are some examples that show what can happen.  First is an extreme case for determining free space on an SD.  This type performance hit will also be true for seek.

Time to determine free space with a properly formatted card using the SdInfo sketch.  See http://forum.arduino.cc/index.php?topic=228128.msg1648437#msg1648437 for more details on code.

Time 0.302 seconds.
Quote

millis: 302
freeClusters: 63883
freeSpace: 2093.32 MB (MB = 1,000,000 bytes)


Same card formatted FAT32 with small clusters on Windows.
34.885 seconds.
Quote

millis: 34885
freeClusters: 4033302
freeSpace: 2065.05 MB (MB = 1,000,000 bytes)


Here are results running the SdFat bench sketch on this card.  Write takes about four times longer with FAT32 and small clusters.

Properly formatted card.
Quote

Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 358.47 KB/sec
Maximum latency: 58888 usec, Minimum Latency: 84 usec, Avg Latency: 273 usec

Starting read test.  Please wait up to a minute
Read 369.19 KB/sec
Maximum latency: 1912 usec, Minimum Latency: 84 usec, Avg Latency: 265 usec


Same card formatted FAT32 with small clusters on Windows.
Quote

Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 89.43 KB/sec
Maximum latency: 117636 usec, Minimum Latency: 84 usec, Avg Latency: 1113 usec

Starting read test.  Please wait up to a minute
Read 225.44 KB/sec
Maximum latency: 1920 usec, Minimum Latency: 80 usec, Avg Latency: 437 usec


These are extreme cases but I do often see this type problem so please use this formatter https://www.sdcard.org/downloads/formatter_4/.  It will always produce the optimum format for your card.

liudr


markallmatt

Wow, OS formatting really isn't very helpful!

keithfarrier


liudr

This also helps with PC reading and writing the same SD card. I have a 32GB SD card that I put in my Dell venue 8 pro. I formatted it in NTFS and the performance of my Dell's video recording was bad. I then reformatted with the formatter and now the video recording is working fine. It's just the SD card performance issue.

pylon

I do use the OS utilities to format an SD card but I use a more flexible OS -> Linux. So your advice is correct for Windows users but Linux users should use the internal command as the SD card association doesn't provide a formatting tool for Linux.

So on Linux users may issue the command:

Code: [Select]
mkfs.vfat -F 32 -s 64 -S 512 /dev/sdb1

given that the card is available as /dev/sdb on the system.
This command does the same as the formatter from the SD card association (use 32kB clusters with a FAT32 file system).

fat16lib

#6
Oct 21, 2014, 12:50 am Last Edit: Oct 22, 2014, 05:02 pm by fat16lib Reason: Add example
Quote
So on Linux users may issue the command:
Linux does not correctly format SD cards.  You can only depend on the SD association's formatter to place file structures correctly to line up with flash chip boundaries.  This is done by placing reserved blocks in the correct places in the file system.

I use Linux but not to format SD cards.

Unfortunately the SD association's standard for formatting SD cards is only available to members.  I was able to access the standard as a consultant and implemented the SdFat formatter example according to this specification.

Here are results of using mkfs.vfat, the SD Association Formatter, and the SdFat formatter
Quote
Formatted with suggested mkfs.vfat command

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973729
freeClusters: 973728
freeSpace: 31907.12 MB (MB = 1,000,000 bytes)
fatStartBlock: 64
fatCount: 2
blocksPerFat: 7616
rootDirStart: 2
dataStartBlock: 15296
Data area is not aligned on flash erase boundaries!
Download and use formatter from www.sdcard.org/consumer!


Formatted with SD Associations formatter

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973584
freeClusters: 973581
freeSpace: 31902.30 MB (MB = 1,000,000 bytes)
fatStartBlock: 9362
fatCount: 2
blocksPerFat: 7607
rootDirStart: 2
dataStartBlock: 24576


Formatted with SdFat formatter

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973584
freeClusters: 973583
freeSpace: 31902.37 MB (MB = 1,000,000 bytes)
fatStartBlock: 9362
fatCount: 2
blocksPerFat: 7607
rootDirStart: 2
dataStartBlock: 24576


I did more tests with Linux and Windows.  I don't have a new version of Mac OS X.

Windows 7/8 does a good job of reformatting a healthy SD card.  Windows just cleans the card and preserves the correct SD format.

Windows 7/8 formats an erased SD as a super floppy but if you choose the correct cluster size, file structures are aligned well.  The SD standard is for an MBR volume so this is not strictly correct.

Mac OS X adds lot of non-standard hidden files and directories.  I have had OS X format an SD card GPT instead of MBR which is unusable on Arduino.  I don't have a machine with a new version of OS X so I can't be sure if Mac OS X still formats SD cards incorrectly.

The mkfs.vfat command suggested by pylon is the worst case.  The card is not MBR and file structures are miss-aligned.  This command will not work for standard SD cards (2 GB or less).

330R

Always format SD cards using SD Formatter https://www.sdcard.org/downloads/formatter_4/.

NEVER format SD card with OS utilities!
Ok, ok, now what if you don't happen to be running Windoze or Max OS? And why should you trust that obscure company/organization, when they themselves declare:

Quote from: www.sdcard.org/legal/index.html
SD Card Association assumes no responsibility regarding the accuracy of the information that is provided by SD Card Association and use of such information is at the recipient's own risk. SD Card Association provides no assurances that any reported problems may be resolved with the use of any information that SD Card Association provides.
while, on the other hand, they charge $2k to become a member?

fat16lib

#8
Jan 31, 2015, 04:40 pm Last Edit: Jan 31, 2015, 04:49 pm by fat16lib
Quote
Ok, ok, now what if you don't happen to be running Windoze or Max OS? And why should you trust that obscure company/organization, when they themselves declare:
Sadly this is the organization that all the big SD manufactures control and your SD card will be manufactured to their standards.

As I said before, the SD association specifies a specialized version of FAT16/FAT32 and now exFAT for SDXC cards that is optimal for the flash layout and chips used in SD cards.

What I don't understand is why Linux does not have a decent formatter for SD cards.  

Linux has tables for handling quirks of USB flash keys and USB hard drives.

If you wish you can use my SdFormatter example in SdFat.  It implements the SD associations standard which is specified in a 70 page Confidential document.  

The document even specifies the MBR and uses it to control reserved space.  You can't just decide on a partition size and location.

Here is an example of one algorithm for FAT12/FAT16 parameters from the standard with proprietary formulas and tables missing.

Quote
Annex: Format Parameter Computations

Data Area should be formatted by the following steps.
1. Sectors per Cluster(SC) is determined from the area size.
2. Number of Root-directory Entries(RDE) is 512.
3. Sector Size(SS) is 512.
4. Reserved Sector Count(RSC) is 1.
5. Total Sectors(TS) is the number of all sectors of the area.
6. FAT bits(12[FAT12], 16[FAT16]) is determined by SC and TS.
7. Sectors per FAT(SF) is computed as following:
   (Standard has a proprietary formula)
  
8. Number of sectors in the system area(SSA) is computed as following:
   (Standard has a proprietary formula)

9. Number of Sectors in Master Boot Record(NOM) is computed as following:
   (Standard has a proprietary formula)

10. If NOM isn't equal to BU, NOM is added BU.
11. Maximum Cluster Number(MAX) is computed as following:
   (Standard has a proprietary formula)

12. Sectors per FAT(SF') is recalculated as following:
   (Standard has a proprietary formula)

13. If SF' isn't equal to SF, SF' is used as SF. And recalculate from step 8.
14. If SF' is equal to SF, parameter computing is complete

Abhijit_Pal

#9
Jul 15, 2015, 05:11 am Last Edit: Jul 15, 2015, 05:21 am by Abhijit_Pal Reason: Found more links
If u r using windows - Make it FAT32 or FAT16. Others dont work.

Found from Arduino SD Lib  :)

https://www.arduino.cc/en/Reference/SD

This is the Link for Sd Formatter  :)

https://www.sdcard.org/downloads/formatter_4/https://www.sdcard.org/downloads/formatter_4/

fat16lib

#10
Jul 18, 2015, 02:15 am Last Edit: Jul 18, 2015, 02:21 am by fat16lib
Quote
If u r using windows - Make it FAT32 or FAT16. Others dont work.
If you format a card with Windows it will not comply with the SD standard.  That's the point of this post.

I repeat, use SD Formatter and avoid potential problems that can be caused by the Windows formatter.

Quote
Found from Arduino SD Lib  :)

https://www.arduino.cc/en/Reference/SD
Much of the write-up for the SD library is wrong.  I wrote the old version of SdFat that is used in SD.h.  Too bad the Arduino company has not fixed bugs in the code or the write-up for years.

Go Up