Don't Format SD cards with OS utilities!

Ok, ok, now what if you don't happen to be running Windoze or Max OS? And why should you trust that obscure company/organization, when they themselves declare:

Sadly this is the organization that all the big SD manufactures control and your SD card will be manufactured to their standards.

As I said before, the SD association specifies a specialized version of FAT16/FAT32 and now exFAT for SDXC cards that is optimal for the flash layout and chips used in SD cards.

What I don't understand is why Linux does not have a decent formatter for SD cards.

Linux has tables for handling quirks of USB flash keys and USB hard drives.

If you wish you can use my SdFormatter example in SdFat. It implements the SD associations standard which is specified in a 70 page Confidential document.

The document even specifies the MBR and uses it to control reserved space. You can't just decide on a partition size and location.

Here is an example of one algorithm for FAT12/FAT16 parameters from the standard with proprietary formulas and tables missing.

Annex: Format Parameter Computations

Data Area should be formatted by the following steps.

  1. Sectors per Cluster(SC) is determined from the area size.

  2. Number of Root-directory Entries(RDE) is 512.

  3. Sector Size(SS) is 512.

  4. Reserved Sector Count(RSC) is 1.

  5. Total Sectors(TS) is the number of all sectors of the area.

  6. FAT bits(12[FAT12], 16[FAT16]) is determined by SC and TS.

  7. Sectors per FAT(SF) is computed as following:
    (Standard has a proprietary formula)

  8. Number of sectors in the system area(SSA) is computed as following:
    (Standard has a proprietary formula)

  9. Number of Sectors in Master Boot Record(NOM) is computed as following:
    (Standard has a proprietary formula)

  10. If NOM isn’t equal to BU, NOM is added BU.

  11. Maximum Cluster Number(MAX) is computed as following:
    (Standard has a proprietary formula)

  12. Sectors per FAT(SF’) is recalculated as following:
    (Standard has a proprietary formula)

  13. If SF’ isn’t equal to SF, SF’ is used as SF. And recalculate from step 8.

  14. If SF’ is equal to SF, parameter computing is complete

1 Like

If u r using windows - Make it FAT32 or FAT16. Others dont work.

Found from Arduino SD Lib :slight_smile:

This is the Link for Sd Formatter :slight_smile:

https://www.sdcard.org/downloads/formatter_4/https://www.sdcard.org/downloads/formatter_4/

If u r using windows - Make it FAT32 or FAT16. Others dont work.

If you format a card with Windows it will not comply with the SD standard. That's the point of this post.

I repeat, use SD Formatter and avoid potential problems that can be caused by the Windows formatter.

Found from Arduino SD Lib :slight_smile:

SD - Arduino Reference

Much of the write-up for the SD library is wrong. I wrote the old version of SdFat that is used in SD.h. Too bad the Arduino company has not fixed bugs in the code or the write-up for years.

Having Win XP/32 and Win7/64 I once used external SD formatting tools and always ran into SDinit/read/write issues for Arduinos (Mega, Due) repeatedly.

Now I'm just using standard Windows format (SD 1GB - 16GB, FAT16/FAT32) - and I'm completely fine with them.

I once used external SD formatting tools and always ran into SDinit/read/write issues for Arduinos (Mega, Due) repeatedly.

Yes there are external formatters that are worse than the Windows formatter.

No tool produces a better result than the SD association formatter since it produces the single format agreed to by the companies that make SD cards.

I wrote the base code for the standard Arduino SD library and it is optimized for the standard format published by the SD association. All versions of my SdFat library are tested with the standard format.

So if you use SD.h or SdFat, you should use the SD association formatter.

using SD formatter 4 I always have issues like before by Windows format:

1 GB SD: SD is not recognized when using SdFat.h

But:
1 GB SD: SD is fine when using SD.h with either formatter,
16 GB SD: SD is fine with either formatter!

what's wrong with 1GB by SdFat.h ??

1 GB SD: SD is not recognized when using SdFat.h

Since you provided almost no information, no output or hardware description, I can't help much.

SD.h is just a wrapper around an old version of SdFat. The main difference is SD.h runs at a slower SPI rate, 4 MHz, vs 8 MHz for newer versions of SdFat.

Often a hardware/SD combination will run at 4 MHz but fail at 8 MHz.

You can set the set the SPI clock rate to 4 MHz with the SdFat begin() call.

sd.begin(chipSelect, SPI_HALF_SPEED);

Please post complete info about you hardware , SD card model, your code, and output with error info.

effing board restriction to 9000 chars does not allow to post complete code.
Code runs on a DUE at most possible speed for the TFT (ILI9341_due lib, SPI clock up to 84 MHz),
for the SD card I didn't alter anything of the pre-settings.

SD cards are from Transcent, 1GB / 16GB, class 10.

#include <SPI.h>

#include <SdFat.h>
SdFat SD;

//#include <SD.h>


// **SNIP**

#define fileIO_OK            +1
#define fileIO_NO_ERR         0
#define fileIO_ERR_CREATE    -1
#define fileIO_ERR_OPEN      -2
#define fileIO_ERR_REMOVE    -3
#define fileIO_ERR_WRITE     -4
#define fileIO_ERR_READ      -5
#define fileIO_ERR_IMPLAUS   -6
#define fileIO_ERR_NAME      -8
#define fileIO_ERR_SDCARD   -16


//=====================================================================================
// SD init
//=====================================================================================


int16_t initSD() {
   char sbuf[128];
   uint32_t  tstamp;
   int16_t   ior=0;
  
   tstamp = clock();
   ior=SD.begin(sd_cs);  // 1==true on success; else 0==false
   while( !ior) {      
      sprintf(sbuf, "#: ...SD not found... ");  
      curlf(); lcdprint("#: ...SD not found... "); 
      Serial.println(sbuf);
      delay(1000);   
      ior=SD.begin(sd_cs);
      if (clock()-tstamp>20000) {Serial.println("#: ...break!"); break; }
   } 
  if(!ior) return fileIO_ERR_SDCARD;     // SD ioresult==0 => error = -16 
  return fileIO_OK ;                     // SD ioresult==1 => ok = 1
} 





//=====================================================================================

void setup() {
   char sbuf[128];   
   int32_t  i=0;
         
   // Serial terminal window
   i=115200;
   Serial.begin(i);  
   Serial.print("Serial started, baud="); Serial.println(i);
   
   
   // TFT LCD
   Serial.println();
   LCDTYPE = _ILI9341_;
   Serial.print("init LCD..."); 

   initLCD(1);   

   Serial.println(" done.");   lcdcls();
   sprintf(sbuf, "LCD=%d wi%dxhi%d Font %dx%d",LCDTYPE,LCDmaxX,LCDmaxY,fontwi,fonthi);
   Serial.println(sbuf); 
   Serial.println();
   lcdcls(); lcdprint(sbuf);
   
   // SD card
   sprintf(sbuf, "SD init... ");   Serial.println(sbuf);
   i = initSD();
   if(i==fileIO_ERR_SDCARD) sprintf(sbuf, "SD failed! ERROR ! ");  
   else sprintf(sbuf, "SD OK ! ");   
   Serial.println(sbuf);   
   curlf();  lcdprint(sbuf);
   
   sprintf(sbuf, "setup(): done.");
   Serial.println(); Serial.println(sbuf);   
   curlf(); curlf(); lcdprint(sbuf);
}



//=====================================================================================
void loop(){
   char     sbuf[128];
  
   

}

SPI clock up to 84 MHz

Max SPI speed for Due is 42 MHz even if you set the clock divider higher.

The SAM3X/A embeds high speed pads able to handle up to 65 MHz for HSMCI and SPI clock lines

So the SPI library has this code.

if (clock < (F_CPU / 255)) {
 div = 255;
 } else if (clock >= (F_CPU / 2)) {
 div = 2;
 } else {
 div = (F_CPU / (clock + 1)) + 1;
 }

On Due you will need to slow SdFat, older SD cards only support 25 MHz max.

I use a clock divider of 2 for newer versions SdFat for a SPI speed of 42 MHz on Due.

SD.h uses a clock divider of 21 for a speed of 4 MHz on Due.

Start with a clock divider of 21 with the 1 GB card and increase it until it fails. SPI_HALF_SPEED has the value 4.

Hi fat16lib,

after having maximized the read rate from within my possibilities to hardly 29.477 kByte/s, which is not even half of what I need, I decided to try the SDFormatter tool. Well, my rate dropped down to 28.587, which may not seem much at all, but is a giant loss regarding my needs.
To get a bit more precise, I have exactly 200,000 µs of time to process 45,056 Bits of data and react to some inputs at the same time, while processing means read, iterate over the bits of each read portion (Bytes atm) and write an output pin accordingly, with a precise timing of 4.4 µs each.

So let's change the scale a bit, here's my measuring code:

#include <SPI.h>
#include <SD.h>         // modified version

File srcFile;

void setup() {
  if (!SD.begin(4, SPI_FULL_SPEED)) {
    return;
  }
  srcFile = SD.open("eight.three");
  if (srcFile) {
    unsigned long t_start = micros();
    for (int i = 0; i < 5632; i++) {
      byte bla = srcFile.read();
    }
      unsigned long t_end = micros();
      Serial.begin(250000);
      while (!Serial) {
      }
      Serial.print(t_end - t_start);
    srcFile.close();
  }
}

void loop() {
}

The output using a "gnome-disks"-fat32-formatted card is: 186588
Same program using the same card, but formatted by SDFormatter4 under Windows (hard inough to manage that ;p ) outputs: 192384
That's a hell of a lot more (recall the 200,000 µs total).

Now here's my actual question: How, if not using a certain formatting tool, will I get this job done in about 100,000 µs?

I actually do need a filesystem btw. I've already had a lot of thoughts about omitting it - finally that's no option. Going raw is a thing I'll definitely try in some other project, though.

Thanks in advance
C1913

Arduino Uno, sorry ^^

C1913,

Your performance problem is not related to format. Single byte read is very slow and SD access has almost nothing to do with read speed. I am not sure why you see a slight difference in read speed, single byte read should be almost independent of the SD or format.

What version of SD.h are you using that accepts "eight.three" as a file name?

You need to read multiple byte into a buffer. You should be able to get a large increase in read speed, at least a factor of ten.

Look at the bench.ino example in SdFat. It can read at up to 400 KB/sec on an Uno.

Edit:

I ran your example with a correctly formatted SD using SdFat and it printed 76140. Not sure why you get such a long time for single byte read.

I ran bench.ino with a 64 byte buffer and got the following

File size 1 MB
Buffer size 64 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
283.37,48164,64,220

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
320.92,2056,68,193

First, thanks for the reply.
It's he SD library that comes with Arduino 1.6.5. The modification I made is to have it accept an optional parameter for SPI speed:
boolean SDClass::begin(uint8_t csPin, uint8_t spiSpeed = SPI_HALF_SPEED)

"eight.three" is just a replacement for the actual file name, a well known 8.3 styled name that I didn't want to be recognized here. Sorry for that.

The bench.ino indeed output something just above 300 kByte/s reading, didn't have time to take a closer look at it, though.
I'll switch to SdFat and try some project related things with it asap. Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.

Allthough it no longer belongs here, for

fat16lib:
Your performance problem is not related to format.

I'm gonna post the results as soon as I'll get some.

Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.

Yes, makes no sense. When I run your code with SdFat I get 76140, almost three time faster. I copied the code and changed SD chip select from 4 to 10 to match my shield.

Edit: I ran the program again and got a longer time. I will investigate more.

Still, the answer for better performance is to do multiple byte reads.

Here is a buffered read that take 16836 micros.

uint8_t buf[64];
    unsigned long t_start = micros();
    
    const size_t TO_READ = 5632;
    size_t nr;
    for (int i = 0; i < TO_READ; i += nr) {
      nr = TO_READ - i;
      if (nr > sizeof(buf)) nr = sizeof(buf);
      size_t ir = srcFile.read(buf, nr);
    //  byte bla = srcFile.read();
      if (ir != nr) {
        // read error
        return;
      }
      for (int j = 0; j < nr; j++) {
        // process bytes here
      }
    }
    unsigned long t_end = micros();

Funny, but I was having endless issues getting an SD card to work on an ethernet shield. After following every suggestion available on the internet, I finally broke down and bought a different shield. Got the Arduino Ethernet shield 2, and popped in the card. STILL didn't read. Argh... Ran across this thread, reformatted the card with the SD formatter, and it started working immediately. Thankfully, it still didn't work in the other shield, so at least the new shield was worth the purchase.

What makes me crazy is that there's this page: SD - Arduino Reference

It says, "Windows : right click on your card’s directory and choose “Format” from the drop down. Make sure you choose FAT as the filesystem."

It's so outdated that it says nothing about FAT32, doesn't say anything about cluster size, and literally says to use the OS formatting. That's what I read long before I found this thread, and it misled me. Someone needs to change it.

Does this apply to usb flash memories as well or just SD cards?

This post is for SD cards.

USB drives used on Arduino are more forgiving than SD cards.

You should format the USB drive with a single MBR based FAT16/FAT32 partition.

Expect poor performance with USB flash drives on Arduino.

I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card! :confused:

There's a license free version of Windows XP for virtual machines you can download for free. For problems like yours I always have a VirtualBox installed with that version.
Might not be the best solution just for formatting SD cards :smiley: bit at least it is one.

I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card!

You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.

fat16lib:
You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.

Thanks! That worked and fast too!