Arduino Forum

Using Arduino => Storage => Topic started by: fat16lib on Mar 24, 2014, 01:53 pm

Title: Don't Format SD cards with OS utilities!
Post by: fat16lib on Mar 24, 2014, 01:53 pm
Always format SD cards using SD Formatter https://www.sdcard.org/downloads/formatter_4/ (https://www.sdcard.org/downloads/formatter_4/).

NEVER format SD card with OS utilities!

Here are some examples that show what can happen.  First is an extreme case for determining free space on an SD.  This type performance hit will also be true for seek.

Time to determine free space with a properly formatted card using the SdInfo sketch.  See http://forum.arduino.cc/index.php?topic=228128.msg1648437#msg1648437 (http://forum.arduino.cc/index.php?topic=228128.msg1648437#msg1648437) for more details on code.

Time 0.302 seconds.
Quote

millis: 302
freeClusters: 63883
freeSpace: 2093.32 MB (MB = 1,000,000 bytes)


Same card formatted FAT32 with small clusters on Windows.
34.885 seconds.
Quote

millis: 34885
freeClusters: 4033302
freeSpace: 2065.05 MB (MB = 1,000,000 bytes)


Here are results running the SdFat bench sketch on this card.  Write takes about four times longer with FAT32 and small clusters.

Properly formatted card.
Quote

Type is FAT16
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 358.47 KB/sec
Maximum latency: 58888 usec, Minimum Latency: 84 usec, Avg Latency: 273 usec

Starting read test.  Please wait up to a minute
Read 369.19 KB/sec
Maximum latency: 1912 usec, Minimum Latency: 84 usec, Avg Latency: 265 usec


Same card formatted FAT32 with small clusters on Windows.
Quote

Type is FAT32
File size 5MB
Buffer size 100 bytes
Starting write test.  Please wait up to a minute
Write 89.43 KB/sec
Maximum latency: 117636 usec, Minimum Latency: 84 usec, Avg Latency: 1113 usec

Starting read test.  Please wait up to a minute
Read 225.44 KB/sec
Maximum latency: 1920 usec, Minimum Latency: 80 usec, Avg Latency: 437 usec


These are extreme cases but I do often see this type problem so please use this formatter https://www.sdcard.org/downloads/formatter_4/ (https://www.sdcard.org/downloads/formatter_4/).  It will always produce the optimum format for your card.
Title: Re: Don't Format SD cards with OS utilities!
Post by: liuzengqiang on Mar 24, 2014, 03:11 pm
Karma Karma!
You saved my day!
Title: Re: Don't Format SD cards with OS utilities!
Post by: markallmatt on May 06, 2014, 06:20 pm
Wow, OS formatting really isn't very helpful!
Title: Re: Don't Format SD cards with OS utilities!
Post by: keithfarrier on May 23, 2014, 12:20 pm
relevant.... thanx for the info...  :)
Title: Re: Don't Format SD cards with OS utilities!
Post by: liuzengqiang on May 23, 2014, 02:30 pm
This also helps with PC reading and writing the same SD card. I have a 32GB SD card that I put in my Dell venue 8 pro. I formatted it in NTFS and the performance of my Dell's video recording was bad. I then reformatted with the formatter and now the video recording is working fine. It's just the SD card performance issue.
Title: Re: Don't Format SD cards with OS utilities!
Post by: pylon on Oct 09, 2014, 06:11 pm
I do use the OS utilities to format an SD card but I use a more flexible OS -> Linux. So your advice is correct for Windows users but Linux users should use the internal command as the SD card association doesn't provide a formatting tool for Linux.

So on Linux users may issue the command:

Code: [Select]
mkfs.vfat -F 32 -s 64 -S 512 /dev/sdb1

given that the card is available as /dev/sdb on the system.
This command does the same as the formatter from the SD card association (use 32kB clusters with a FAT32 file system).
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Oct 21, 2014, 12:50 am
Quote
So on Linux users may issue the command:
Linux does not correctly format SD cards.  You can only depend on the SD association's formatter to place file structures correctly to line up with flash chip boundaries.  This is done by placing reserved blocks in the correct places in the file system.

I use Linux but not to format SD cards.

Unfortunately the SD association's standard for formatting SD cards is only available to members.  I was able to access the standard as a consultant and implemented the SdFat formatter example according to this specification.

Here are results of using mkfs.vfat, the SD Association Formatter, and the SdFat formatter
Quote
Formatted with suggested mkfs.vfat command

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973729
freeClusters: 973728
freeSpace: 31907.12 MB (MB = 1,000,000 bytes)
fatStartBlock: 64
fatCount: 2
blocksPerFat: 7616
rootDirStart: 2
dataStartBlock: 15296
Data area is not aligned on flash erase boundaries!
Download and use formatter from www.sdcard.org/consumer!


Formatted with SD Associations formatter

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973584
freeClusters: 973581
freeSpace: 31902.30 MB (MB = 1,000,000 bytes)
fatStartBlock: 9362
fatCount: 2
blocksPerFat: 7607
rootDirStart: 2
dataStartBlock: 24576


Formatted with SdFat formatter

Volume is FAT32
blocksPerCluster: 64
clusterCount: 973584
freeClusters: 973583
freeSpace: 31902.37 MB (MB = 1,000,000 bytes)
fatStartBlock: 9362
fatCount: 2
blocksPerFat: 7607
rootDirStart: 2
dataStartBlock: 24576


I did more tests with Linux and Windows.  I don't have a new version of Mac OS X.

Windows 7/8 does a good job of reformatting a healthy SD card.  Windows just cleans the card and preserves the correct SD format.

Windows 7/8 formats an erased SD as a super floppy but if you choose the correct cluster size, file structures are aligned well.  The SD standard is for an MBR volume so this is not strictly correct.

Mac OS X adds lot of non-standard hidden files and directories.  I have had OS X format an SD card GPT instead of MBR which is unusable on Arduino.  I don't have a machine with a new version of OS X so I can't be sure if Mac OS X still formats SD cards incorrectly.

The mkfs.vfat command suggested by pylon is the worst case.  The card is not MBR and file structures are miss-aligned.  This command will not work for standard SD cards (2 GB or less).
Title: Re: Don't Format SD cards with OS utilities!
Post by: 330R on Jan 31, 2015, 11:17 am
Always format SD cards using SD Formatter https://www.sdcard.org/downloads/formatter_4/ (https://www.sdcard.org/downloads/formatter_4/).

NEVER format SD card with OS utilities!
Ok, ok, now what if you don't happen to be running Windoze or Max OS? And why should you trust that obscure company/organization, when they themselves declare:

Quote from: www.sdcard.org/legal/index.html
SD Card Association assumes no responsibility regarding the accuracy of the information that is provided by SD Card Association and use of such information is at the recipient's own risk. SD Card Association provides no assurances that any reported problems may be resolved with the use of any information that SD Card Association provides.
while, on the other hand, they charge $2k to become a member?
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Jan 31, 2015, 04:40 pm
Quote
Ok, ok, now what if you don't happen to be running Windoze or Max OS? And why should you trust that obscure company/organization, when they themselves declare:
Sadly this is the organization that all the big SD manufactures control and your SD card will be manufactured to their standards.

As I said before, the SD association specifies a specialized version of FAT16/FAT32 and now exFAT for SDXC cards that is optimal for the flash layout and chips used in SD cards.

What I don't understand is why Linux does not have a decent formatter for SD cards.  

Linux has tables for handling quirks of USB flash keys and USB hard drives.

If you wish you can use my SdFormatter example in SdFat.  It implements the SD associations standard which is specified in a 70 page Confidential document.  

The document even specifies the MBR and uses it to control reserved space.  You can't just decide on a partition size and location.

Here is an example of one algorithm for FAT12/FAT16 parameters from the standard with proprietary formulas and tables missing.

Quote
Annex: Format Parameter Computations

Data Area should be formatted by the following steps.
1. Sectors per Cluster(SC) is determined from the area size.
2. Number of Root-directory Entries(RDE) is 512.
3. Sector Size(SS) is 512.
4. Reserved Sector Count(RSC) is 1.
5. Total Sectors(TS) is the number of all sectors of the area.
6. FAT bits(12[FAT12], 16[FAT16]) is determined by SC and TS.
7. Sectors per FAT(SF) is computed as following:
   (Standard has a proprietary formula)
  
8. Number of sectors in the system area(SSA) is computed as following:
   (Standard has a proprietary formula)

9. Number of Sectors in Master Boot Record(NOM) is computed as following:
   (Standard has a proprietary formula)

10. If NOM isn't equal to BU, NOM is added BU.
11. Maximum Cluster Number(MAX) is computed as following:
   (Standard has a proprietary formula)

12. Sectors per FAT(SF') is recalculated as following:
   (Standard has a proprietary formula)

13. If SF' isn't equal to SF, SF' is used as SF. And recalculate from step 8.
14. If SF' is equal to SF, parameter computing is complete
Title: Re: Don't Format SD cards with OS utilities!
Post by: Abhijit_Pal on Jul 15, 2015, 05:11 am
If u r using windows - Make it FAT32 or FAT16. Others dont work.

Found from Arduino SD Lib  :)

https://www.arduino.cc/en/Reference/SD (https://www.arduino.cc/en/Reference/SD)

This is the Link for Sd Formatter  :)

https://www.sdcard.org/downloads/formatter_4/https://www.sdcard.org/downloads/formatter_4/ (https://www.sdcard.org/downloads/formatter_4/https://www.sdcard.org/downloads/formatter_4/)
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Jul 18, 2015, 02:15 am
Quote
If u r using windows - Make it FAT32 or FAT16. Others dont work.
If you format a card with Windows it will not comply with the SD standard.  That's the point of this post.

I repeat, use SD Formatter and avoid potential problems that can be caused by the Windows formatter.

Quote
Found from Arduino SD Lib  :)

https://www.arduino.cc/en/Reference/SD
Much of the write-up for the SD library is wrong.  I wrote the old version of SdFat that is used in SD.h.  Too bad the Arduino company has not fixed bugs in the code or the write-up for years.
Title: Re: Don't Format SD cards with OS utilities!
Post by: ArthurD on Sep 12, 2015, 10:48 am
Having Win XP/32 and Win7/64 I once used external SD formatting tools and always ran into SDinit/read/write issues for Arduinos (Mega, Due) repeatedly.

Now I'm just using standard Windows format (SD 1GB - 16GB, FAT16/FAT32)  - and I'm completely fine with them.
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Sep 13, 2015, 04:32 am
Quote
I once used external SD formatting tools and always ran into SDinit/read/write issues for Arduinos (Mega, Due) repeatedly.
Yes there are external formatters that are worse than the Windows formatter.

No tool produces a better result than the SD association formatter since it produces the single format agreed to by the companies that make SD cards.

I wrote the base code for the standard Arduino SD library and it is optimized for the standard format published by the SD association.  All versions of my SdFat library are tested with the standard format.

So if you use SD.h or SdFat, you should use the SD association formatter.
Title: Re: Don't Format SD cards with OS utilities!
Post by: ArthurD on Sep 28, 2015, 06:03 pm
using SD formatter 4 I always have issues like before by Windows format:

1 GB SD: SD is not recognized when using SdFat.h

But:
1 GB SD: SD is fine when using SD.h with either formatter, 
16 GB SD: SD is fine with either formatter!

what's wrong with 1GB by SdFat.h ??
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Sep 29, 2015, 12:33 am
Quote
1 GB SD: SD is not recognized when using SdFat.h
Since you provided almost no information, no output or hardware description, I can't help much.

SD.h is just a wrapper around an old version of SdFat.  The main difference is SD.h runs at a slower SPI rate, 4 MHz, vs 8 MHz for newer versions of SdFat.

Often a hardware/SD combination will run at 4 MHz but fail at 8 MHz.

You can set the set the SPI clock rate to 4 MHz with the SdFat begin() call.
Quote
sd.begin(chipSelect, SPI_HALF_SPEED);
Please post complete info about you hardware , SD card model, your code, and output with error info.
Title: Re: Don't Format SD cards with OS utilities!
Post by: ArthurD on Sep 29, 2015, 09:10 am
effing board restriction to 9000 chars does not allow to post complete code.
Code runs on a DUE at most possible speed for the TFT (ILI9341_due lib, SPI clock up to 84 MHz),
for the SD card I didn't alter anything of the pre-settings.

SD cards are from Transcent, 1GB / 16GB, class 10.

Code: [Select]

#include <SPI.h>

#include <SdFat.h>
SdFat SD;

//#include <SD.h>


// **SNIP**

#define fileIO_OK            +1
#define fileIO_NO_ERR         0
#define fileIO_ERR_CREATE    -1
#define fileIO_ERR_OPEN      -2
#define fileIO_ERR_REMOVE    -3
#define fileIO_ERR_WRITE     -4
#define fileIO_ERR_READ      -5
#define fileIO_ERR_IMPLAUS   -6
#define fileIO_ERR_NAME      -8
#define fileIO_ERR_SDCARD   -16


//=====================================================================================
// SD init
//=====================================================================================


int16_t initSD() {
   char sbuf[128];
   uint32_t  tstamp;
   int16_t   ior=0;
 
   tstamp = clock();
   ior=SD.begin(sd_cs);  // 1==true on success; else 0==false
   while( !ior) {     
      sprintf(sbuf, "#: ...SD not found... "); 
      curlf(); lcdprint("#: ...SD not found... ");
      Serial.println(sbuf);
      delay(1000);   
      ior=SD.begin(sd_cs);
      if (clock()-tstamp>20000) {Serial.println("#: ...break!"); break; }
   }
  if(!ior) return fileIO_ERR_SDCARD;     // SD ioresult==0 => error = -16
  return fileIO_OK ;                     // SD ioresult==1 => ok = 1
}





//=====================================================================================

void setup() {
   char sbuf[128];   
   int32_t  i=0;
         
   // Serial terminal window
   i=115200;
   Serial.begin(i); 
   Serial.print("Serial started, baud="); Serial.println(i);
   
   
   // TFT LCD
   Serial.println();
   LCDTYPE = _ILI9341_;
   Serial.print("init LCD...");

   initLCD(1);   

   Serial.println(" done.");   lcdcls();
   sprintf(sbuf, "LCD=%d wi%dxhi%d Font %dx%d",LCDTYPE,LCDmaxX,LCDmaxY,fontwi,fonthi);
   Serial.println(sbuf);
   Serial.println();
   lcdcls(); lcdprint(sbuf);
   
   // SD card
   sprintf(sbuf, "SD init... ");   Serial.println(sbuf);
   i = initSD();
   if(i==fileIO_ERR_SDCARD) sprintf(sbuf, "SD failed! ERROR ! "); 
   else sprintf(sbuf, "SD OK ! ");   
   Serial.println(sbuf);   
   curlf();  lcdprint(sbuf);
   
   sprintf(sbuf, "setup(): done.");
   Serial.println(); Serial.println(sbuf);   
   curlf(); curlf(); lcdprint(sbuf);
}



//=====================================================================================
void loop(){
   char     sbuf[128];
 
   

}



Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Sep 29, 2015, 08:38 pm
Quote
SPI clock up to 84 MHz
Max SPI speed for Due is 42 MHz even if you set the clock divider higher.

Quote
The SAM3X/A embeds high speed pads able to handle up to 65 MHz for HSMCI and SPI clock lines
So the SPI library has this code.

Code: [Select]
if (clock < (F_CPU / 255)) {
 div = 255;
 } else if (clock >= (F_CPU / 2)) {
 div = 2;
 } else {
 div = (F_CPU / (clock + 1)) + 1;
 }


On Due you will need to slow SdFat,  older SD cards only support 25 MHz max.  

I use a clock divider of 2 for newer versions SdFat for a SPI speed of 42 MHz on Due.

SD.h uses a clock divider of 21 for a speed of 4 MHz on Due.

Start with a clock divider of 21 with the 1 GB card and increase it until it fails.  SPI_HALF_SPEED has the value 4.
Title: Re: Don't Format SD cards with OS utilities!
Post by: C1913 on Oct 08, 2015, 12:26 am
Hi fat16lib,

after having maximized the read rate from within my possibilities to hardly 29.477 kByte/s, which is not even half of what I need, I decided to try the SDFormatter tool. Well, my rate dropped down to 28.587, which may not seem much at all, but is a giant loss regarding my needs.
To get a bit more precise, I have exactly 200,000 µs of time to process 45,056 Bits of data and react to some inputs at the same time, while processing means read, iterate over the bits of each read portion (Bytes atm) and write an output pin accordingly, with a precise timing of 4.4 µs each.

So let's change the scale a bit, here's my measuring code:

Code: [Select]
#include <SPI.h>
#include <SD.h>         // modified version

File srcFile;

void setup() {
  if (!SD.begin(4, SPI_FULL_SPEED)) {
    return;
  }
  srcFile = SD.open("eight.three");
  if (srcFile) {
    unsigned long t_start = micros();
    for (int i = 0; i < 5632; i++) {
      byte bla = srcFile.read();
    }
      unsigned long t_end = micros();
      Serial.begin(250000);
      while (!Serial) {
      }
      Serial.print(t_end - t_start);
    srcFile.close();
  }
}

void loop() {
}


The output using a "gnome-disks"-fat32-formatted card is: 186588
Same program using the same card, but formatted by SDFormatter4 under Windows (hard inough to manage that ;p ) outputs: 192384
That's a hell of a lot more (recall the 200,000 µs total).


Now here's my actual question: How, if not using a certain formatting tool, will I get this job done in about 100,000 µs?

I actually do need a filesystem btw. I've already had a lot of thoughts about omitting it - finally that's no option. Going raw is a thing I'll definitely try in some other project, though.

Thanks in advance
  C1913

Arduino Uno, sorry ^^
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Oct 08, 2015, 05:13 am
C1913,

Your performance problem is not related to format.  Single byte read is very slow and SD access has almost nothing to do with read speed.  I am not sure why you see a slight difference in read speed, single byte read should be almost independent of the SD or format.

What version of SD.h are you using that accepts "eight.three" as a file name?

You need to read multiple byte into a buffer.  You should be able to get a large increase in read speed, at least a factor of ten.

Look at the bench.ino example in SdFat.  It can read at up to 400 KB/sec on an Uno.

Edit:

I ran your example with a correctly formatted SD using SdFat and it printed 76140.  Not sure why you get such a long time for single byte read.

I ran bench.ino with a 64 byte buffer and got the following
Quote
File size 1 MB
Buffer size 64 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
283.37,48164,64,220

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
320.92,2056,68,193
Title: Re: Don't Format SD cards with OS utilities!
Post by: C1913 on Oct 08, 2015, 07:31 pm
First, thanks for the reply.
It's he SD library that comes with Arduino 1.6.5. The modification I made is to have it accept an optional parameter for SPI speed:
boolean SDClass::begin(uint8_t csPin, uint8_t spiSpeed = SPI_HALF_SPEED)

"eight.three" is just a replacement for the actual file name, a well known 8.3 styled name that I didn't want to be recognized here. Sorry for that.

The bench.ino indeed output something just above 300 kByte/s reading, didn't have time to take a closer look at it, though.
I'll switch to SdFat and try some project related things with it asap. Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.


Allthough it no longer belongs here, for
Your performance problem is not related to format.
I'm gonna post the results as soon as I'll get some.
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Oct 09, 2015, 12:40 am
Quote
Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.
Yes, makes no sense.  When I run your code with SdFat I get 76140, almost three time faster.  I copied the code and changed SD chip select from 4 to 10 to match my shield.

Edit: I ran the program again and got a longer time.  I will investigate more.

Still, the answer for better performance is to do multiple byte reads.

Here is a buffered read that take 16836 micros.

Code: [Select]

uint8_t buf[64];
    unsigned long t_start = micros();
    
    const size_t TO_READ = 5632;
    size_t nr;
    for (int i = 0; i < TO_READ; i += nr) {
      nr = TO_READ - i;
      if (nr > sizeof(buf)) nr = sizeof(buf);
      size_t ir = srcFile.read(buf, nr);
    //  byte bla = srcFile.read();
      if (ir != nr) {
        // read error
        return;
      }
      for (int j = 0; j < nr; j++) {
        // process bytes here
      }
    }
    unsigned long t_end = micros();
Title: Re: Don't Format SD cards with OS utilities!
Post by: bitNine on Oct 09, 2015, 05:01 pm
Funny, but I was having endless issues getting an SD card to work on an ethernet shield. After following every suggestion available on the internet, I finally broke down and bought a different shield. Got the Arduino Ethernet shield 2, and popped in the card. STILL didn't read. Argh... Ran across this thread, reformatted the card with the SD formatter, and it started working immediately. Thankfully, it still didn't work in the other shield, so at least the new shield was worth the purchase.

What makes me crazy is that there's this page: https://www.arduino.cc/en/Reference/SDCardNotes

It says, "Windows : right click on your card's directory and choose "Format" from the drop down. Make sure you choose FAT as the filesystem."

It's so outdated that it says nothing about FAT32, doesn't say anything about cluster size, and literally says to use the OS formatting. That's what I read long before I found this thread, and it misled me. Someone needs to change it.
Title: Re: Don't Format SD cards with OS utilities!
Post by: Skylark on Oct 24, 2015, 08:32 pm
Does this apply to usb flash memories as well or just SD cards?
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Oct 26, 2015, 10:24 pm
This post is for SD cards.

USB drives used on Arduino are more forgiving than SD cards. 

You should format the USB drive with a single MBR based FAT16/FAT32 partition.

Expect poor performance with USB flash drives on Arduino.
Title: Re: Don't Format SD cards with OS utilities!
Post by: NattyFido on Oct 30, 2015, 04:40 pm
I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card! :smiley-confuse:
Title: Re: Don't Format SD cards with OS utilities!
Post by: C1913 on Oct 30, 2015, 05:18 pm
There's a license free version of Windows XP for virtual machines you can download for free. For problems like yours I always have a VirtualBox installed with that version.
Might not be the best solution just for formatting SD cards   :D bit at least it is one.
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Oct 31, 2015, 03:29 am
Quote
I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card!
You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.
Title: Re: Don't Format SD cards with OS utilities!
Post by: NattyFido on Nov 03, 2015, 07:56 pm
You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.
Thanks! That worked and fast too!
Title: Re: Don't Format SD cards with OS utilities!
Post by: pankaj554 on Dec 10, 2015, 11:44 am
 If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Dec 10, 2015, 02:16 pm
Quote
If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.
Please read before you post.  The format won't be optimal for Arduino if you use OS utilities.  Arduino is like not Windows, OS X, or Linux.  Desktop OSs have huge amounts of RAM for buffering so the format is less important.

I suggest you learn about the internals of SD cards and how FAT16/FAT32 is implemented for Arduino before providing advice.
Title: Re: Don't Format SD cards with OS utilities!
Post by: Flinkly on Jul 22, 2016, 12:33 am
Thanks for this, and SdFat.  I'm still working on buffering (making sense of it, ME here...), but whenever I have a question about Arduino's and SDs, and ask the Google, you (in some flavor) come up and answer my question and more.

If only SDIO didn't seem so much more intimidating than SPI (which is so simple!).

Thanks!
Title: Re: Don't Format SD cards with OS utilities!
Post by: Eimerkette on Mar 12, 2017, 10:18 am
Thanks for the information about using "SDFormatter V4.0" to format SD card.

Because of having no success to get my "Arduino UNO" - "SD card" running, I searched for posts which describes the same problem. But I didn't found any solution/hint which will help. In the meantime I spend many hours/days...

Therefore I also tried to use "SDFormatter V4.0" to format my SD card, but I did not solve my problem.

My "Arduino UNO" - "SD card" is still not running. That's really frustrating and makes no fun.

The best result (using "CardInfo") was:

Code: [Select]

Initializing SD card...Wiring is correct and a card is present.

Card type: SDHC
Could not find FAT16/FAT32 partition.
Make sure you've formatted the card


but it is hard to reproduce this situation.

Any hints?
Title: Re: Don't Format SD cards with OS utilities!
Post by: _pepe_ on Mar 12, 2017, 03:08 pm
Supprimé
Title: Re: Don't Format SD cards with OS utilities!
Post by: Eimerkette on Mar 12, 2017, 05:15 pm
Hi _pepe_,

thanks for this hint.

I'll do it and report the corresponding error codes in a new topic (http://forum.arduino.cc/index.php?topic=462102.0).
Title: Re: Don't Format SD cards with OS utilities!
Post by: dlchambers on Jun 26, 2017, 09:18 pm
Unfortunately, this utility formats the SD as FAT32, but the Arduino SD lib wants FAT16. So the tool is useless.
Title: Re: Don't Format SD cards with OS utilities!
Post by: C1913 on Jun 26, 2017, 10:08 pm
Quote
So the tool is useless
So are cars, if I dont know how to drive them. Or foreign languages I don't speak. They are indeed useless, but to me, not in general.
Only cause you're frustrated is no reason to be rude. People here are there to help you, not to bother you with useless crap   ;)
Title: Re: Don't Format SD cards with OS utilities!
Post by: fat16lib on Jul 01, 2017, 06:17 pm
Quote
Unfortunately, this utility formats the SD as FAT32, but the Arduino SD lib wants FAT16. So the tool is useless.
SD cards are designed to be formatted with a layout that matches the card's flash chips.  SDFormatter produces the optimal layout by inserting hidden sectors to adjust the layout to chip boundaries.  SDFormatter chooses an appropriate cluster size.  There are no options for the standard format.

SDFormatter uses FAT12 for very small cards.  FAT16 is used for cards <= 2GB, FAT32 for cards <= 32 GB, and exFAT for cards larger than 32GB.

The SD library supports FAT16/FAT32 and is optimized for the layout SDFormatter produces.
Title: Re: Don't Format SD cards with OS utilities!
Post by: soundlord on Feb 06, 2018, 03:44 pm
I do use the OS utilities to format an SD card but I use a more flexible OS -> Linux. So your advice is correct for Windows users but Linux users should use the internal command as the SD card association doesn't provide a formatting tool for Linux.

So on Linux users may issue the command:

Code: [Select]
mkfs.vfat -F 32 -s 64 -S 512 /dev/sdb1

given that the card is available as /dev/sdb on the system.
This command does the same as the formatter from the SD card association (use 32kB clusters with a FAT32 file system).
Never worked for me... SD.begin(cable select slot) returns 0.

SOLUTION:

In fact you have to (using Linux):

1. have root rights  (su root)
2. in bash (shell) --> msdosfs -I /dev/sdd (assuming /dev/sdd is the sdcard device)
3. in arduino 1.8.5 --> verify you installed the SdFAT library
Use the example called SdFormatter
4. Change the cable select port to 4 for the arduino with LCD module with SD card reader
5. Load the program in the arduino and follow the instructions using the serial monitor




 
Title: Re: Don't Format SD cards with OS utilities!
Post by: afedorov on Mar 03, 2018, 07:33 pm
I see strange behavior with 32G sd card. Sometimes it initializes some times it doesn't. Can it happen becauseI used MacOs to format the card?
Title: Re: Don't Format SD cards with OS utilities!
Post by: apollinaire on May 31, 2018, 01:17 pm
Has anybody yet tried formatting the SD card in a digital camera as a workaround?
I can't test it myself (no SD.shield in my drawer), just an idea...
Title: Re: Don't Format SD cards with OS utilities!
Post by: capitkle on Feb 02, 2019, 06:46 am
Hello everyone,

I am trying to use an SD card with my UNO for the first time and I'm running into a lot of the same problems that other posters have experienced. Unfortunately, I have not had any success with the posted solutions.

Here are some of the specs for my setup:
Board: Kuman UNO
SD Breakout: velleman VMA304
Arduino IDE: 1.8.5
SD Library version 1.1.1
SdFat version 1.0.16

I first tried running the built-in ReadWrite tutorial, but the SD card initialization failed. I then formatted the SD card with the SD Card Formatter version 5.0.1, from the SD Association's website, but still the initialization failed.

After that, I downloaded the SdFat library and tried running the SdInfo example. It was able to pull some of the info from the card, but still failed to initialize. The last thing I tried was formatting the card again using the SdFormatter in the SdFat library, but the serial monitor displayed the following error after starting the formatter:

.........
Enter option: F
Card Size: 15552 MB,

Erasing
.error: erase failed
SD error: 5,4f
.........

Is there anyone who has seen or had experience with this error before, or who could possibly point me in the right direction to solve this issue?

Thank you,