Don't Format SD cards with OS utilities!

First, thanks for the reply.
It's he SD library that comes with Arduino 1.6.5. The modification I made is to have it accept an optional parameter for SPI speed:
boolean SDClass::begin(uint8_t csPin, uint8_t spiSpeed = SPI_HALF_SPEED)

"eight.three" is just a replacement for the actual file name, a well known 8.3 styled name that I didn't want to be recognized here. Sorry for that.

The bench.ino indeed output something just above 300 kByte/s reading, didn't have time to take a closer look at it, though.
I'll switch to SdFat and try some project related things with it asap. Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.

Allthough it no longer belongs here, for

fat16lib:
Your performance problem is not related to format.

I'm gonna post the results as soon as I'll get some.

Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.

Yes, makes no sense. When I run your code with SdFat I get 76140, almost three time faster. I copied the code and changed SD chip select from 4 to 10 to match my shield.

Edit: I ran the program again and got a longer time. I will investigate more.

Still, the answer for better performance is to do multiple byte reads.

Here is a buffered read that take 16836 micros.

uint8_t buf[64];
    unsigned long t_start = micros();
    
    const size_t TO_READ = 5632;
    size_t nr;
    for (int i = 0; i < TO_READ; i += nr) {
      nr = TO_READ - i;
      if (nr > sizeof(buf)) nr = sizeof(buf);
      size_t ir = srcFile.read(buf, nr);
    //  byte bla = srcFile.read();
      if (ir != nr) {
        // read error
        return;
      }
      for (int j = 0; j < nr; j++) {
        // process bytes here
      }
    }
    unsigned long t_end = micros();

Funny, but I was having endless issues getting an SD card to work on an ethernet shield. After following every suggestion available on the internet, I finally broke down and bought a different shield. Got the Arduino Ethernet shield 2, and popped in the card. STILL didn't read. Argh... Ran across this thread, reformatted the card with the SD formatter, and it started working immediately. Thankfully, it still didn't work in the other shield, so at least the new shield was worth the purchase.

What makes me crazy is that there's this page: SD - Arduino Reference

It says, "Windows : right click on your card’s directory and choose “Format” from the drop down. Make sure you choose FAT as the filesystem."

It's so outdated that it says nothing about FAT32, doesn't say anything about cluster size, and literally says to use the OS formatting. That's what I read long before I found this thread, and it misled me. Someone needs to change it.

Does this apply to usb flash memories as well or just SD cards?

This post is for SD cards.

USB drives used on Arduino are more forgiving than SD cards.

You should format the USB drive with a single MBR based FAT16/FAT32 partition.

Expect poor performance with USB flash drives on Arduino.

I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card! :confused:

There's a license free version of Windows XP for virtual machines you can download for free. For problems like yours I always have a VirtualBox installed with that version.
Might not be the best solution just for formatting SD cards :smiley: bit at least it is one.

I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card!

You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.

fat16lib:
You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.

Thanks! That worked and fast too!

If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.

If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.

Please read before you post. The format won't be optimal for Arduino if you use OS utilities. Arduino is like not Windows, OS X, or Linux. Desktop OSs have huge amounts of RAM for buffering so the format is less important.

I suggest you learn about the internals of SD cards and how FAT16/FAT32 is implemented for Arduino before providing advice.

Thanks for this, and SdFat. I'm still working on buffering (making sense of it, ME here...), but whenever I have a question about Arduino's and SDs, and ask the Google, you (in some flavor) come up and answer my question and more.

If only SDIO didn't seem so much more intimidating than SPI (which is so simple!).

Thanks!

Thanks for the information about using "SDFormatter V4.0" to format SD card.

Because of having no success to get my "Arduino UNO" - "SD card" running, I searched for posts which describes the same problem. But I didn't found any solution/hint which will help. In the meantime I spend many hours/days...

Therefore I also tried to use "SDFormatter V4.0" to format my SD card, but I did not solve my problem.

My "Arduino UNO" - "SD card" is still not running. That's really frustrating and makes no fun.

The best result (using "CardInfo") was:

Initializing SD card...Wiring is correct and a card is present.

Card type: SDHC
Could not find FAT16/FAT32 partition.
Make sure you've formatted the card

but it is hard to reproduce this situation.

Any hints?

Hi pepe,

thanks for this hint.

I'll do it and report the corresponding error codes in a new topic.

Unfortunately, this utility formats the SD as FAT32, but the Arduino SD lib wants FAT16. So the tool is useless.

So the tool is useless

So are cars, if I dont know how to drive them. Or foreign languages I don't speak. They are indeed useless, but to me, not in general.
Only cause you're frustrated is no reason to be rude. People here are there to help you, not to bother you with useless crap :wink:

Unfortunately, this utility formats the SD as FAT32, but the Arduino SD lib wants FAT16. So the tool is useless.

SD cards are designed to be formatted with a layout that matches the card's flash chips. SDFormatter produces the optimal layout by inserting hidden sectors to adjust the layout to chip boundaries. SDFormatter chooses an appropriate cluster size. There are no options for the standard format.

SDFormatter uses FAT12 for very small cards. FAT16 is used for cards <= 2GB, FAT32 for cards <= 32 GB, and exFAT for cards larger than 32GB.

The SD library supports FAT16/FAT32 and is optimized for the layout SDFormatter produces.

pylon:
I do use the OS utilities to format an SD card but I use a more flexible OS -> Linux. So your advice is correct for Windows users but Linux users should use the internal command as the SD card association doesn't provide a formatting tool for Linux.

So on Linux users may issue the command:

mkfs.vfat -F 32 -s 64 -S 512 /dev/sdb1

given that the card is available as /dev/sdb on the system.
This command does the same as the formatter from the SD card association (use 32kB clusters with a FAT32 file system).

Never worked for me... SD.begin(cable select slot) returns 0.

SOLUTION:

In fact you have to (using Linux):

  1. have root rights (su root)
  2. in bash (shell) --> msdosfs -I /dev/sdd (assuming /dev/sdd is the sdcard device)
  3. in arduino 1.8.5 --> verify you installed the SdFAT library
    Use the example called SdFormatter
  4. Change the cable select port to 4 for the arduino with LCD module with SD card reader
  5. Load the program in the arduino and follow the instructions using the serial monitor

I see strange behavior with 32G sd card. Sometimes it initializes some times it doesn't. Can it happen becauseI used MacOs to format the card?

Has anybody yet tried formatting the SD card in a digital camera as a workaround?
I can't test it myself (no SD.shield in my drawer), just an idea...