Go Down

Topic: Don't Format SD cards with OS utilities! (Read 71385 times) previous topic - next topic

ArthurD

#15
Sep 29, 2015, 09:10 am Last Edit: Sep 29, 2015, 09:55 am by ArthurD
effing board restriction to 9000 chars does not allow to post complete code.
Code runs on a DUE at most possible speed for the TFT (ILI9341_due lib, SPI clock up to 84 MHz),
for the SD card I didn't alter anything of the pre-settings.

SD cards are from Transcent, 1GB / 16GB, class 10.

Code: [Select]

#include <SPI.h>

#include <SdFat.h>
SdFat SD;

//#include <SD.h>


// **SNIP**

#define fileIO_OK            +1
#define fileIO_NO_ERR         0
#define fileIO_ERR_CREATE    -1
#define fileIO_ERR_OPEN      -2
#define fileIO_ERR_REMOVE    -3
#define fileIO_ERR_WRITE     -4
#define fileIO_ERR_READ      -5
#define fileIO_ERR_IMPLAUS   -6
#define fileIO_ERR_NAME      -8
#define fileIO_ERR_SDCARD   -16


//=====================================================================================
// SD init
//=====================================================================================


int16_t initSD() {
   char sbuf[128];
   uint32_t  tstamp;
   int16_t   ior=0;
 
   tstamp = clock();
   ior=SD.begin(sd_cs);  // 1==true on success; else 0==false
   while( !ior) {     
      sprintf(sbuf, "#: ...SD not found... "); 
      curlf(); lcdprint("#: ...SD not found... ");
      Serial.println(sbuf);
      delay(1000);   
      ior=SD.begin(sd_cs);
      if (clock()-tstamp>20000) {Serial.println("#: ...break!"); break; }
   }
  if(!ior) return fileIO_ERR_SDCARD;     // SD ioresult==0 => error = -16
  return fileIO_OK ;                     // SD ioresult==1 => ok = 1
}





//=====================================================================================

void setup() {
   char sbuf[128];   
   int32_t  i=0;
         
   // Serial terminal window
   i=115200;
   Serial.begin(i); 
   Serial.print("Serial started, baud="); Serial.println(i);
   
   
   // TFT LCD
   Serial.println();
   LCDTYPE = _ILI9341_;
   Serial.print("init LCD...");

   initLCD(1);   

   Serial.println(" done.");   lcdcls();
   sprintf(sbuf, "LCD=%d wi%dxhi%d Font %dx%d",LCDTYPE,LCDmaxX,LCDmaxY,fontwi,fonthi);
   Serial.println(sbuf);
   Serial.println();
   lcdcls(); lcdprint(sbuf);
   
   // SD card
   sprintf(sbuf, "SD init... ");   Serial.println(sbuf);
   i = initSD();
   if(i==fileIO_ERR_SDCARD) sprintf(sbuf, "SD failed! ERROR ! "); 
   else sprintf(sbuf, "SD OK ! ");   
   Serial.println(sbuf);   
   curlf();  lcdprint(sbuf);
   
   sprintf(sbuf, "setup(): done.");
   Serial.println(); Serial.println(sbuf);   
   curlf(); curlf(); lcdprint(sbuf);
}



//=====================================================================================
void loop(){
   char     sbuf[128];
 
   

}




fat16lib

#16
Sep 29, 2015, 08:38 pm Last Edit: Sep 29, 2015, 09:07 pm by fat16lib
Quote
SPI clock up to 84 MHz
Max SPI speed for Due is 42 MHz even if you set the clock divider higher.

Quote
The SAM3X/A embeds high speed pads able to handle up to 65 MHz for HSMCI and SPI clock lines
So the SPI library has this code.

Code: [Select]
if (clock < (F_CPU / 255)) {
 div = 255;
 } else if (clock >= (F_CPU / 2)) {
 div = 2;
 } else {
 div = (F_CPU / (clock + 1)) + 1;
 }


On Due you will need to slow SdFat,  older SD cards only support 25 MHz max.  

I use a clock divider of 2 for newer versions SdFat for a SPI speed of 42 MHz on Due.

SD.h uses a clock divider of 21 for a speed of 4 MHz on Due.

Start with a clock divider of 21 with the 1 GB card and increase it until it fails.  SPI_HALF_SPEED has the value 4.

C1913

#17
Oct 08, 2015, 12:26 am Last Edit: Oct 08, 2015, 12:46 am by C1913
Hi fat16lib,

after having maximized the read rate from within my possibilities to hardly 29.477 kByte/s, which is not even half of what I need, I decided to try the SDFormatter tool. Well, my rate dropped down to 28.587, which may not seem much at all, but is a giant loss regarding my needs.
To get a bit more precise, I have exactly 200,000 µs of time to process 45,056 Bits of data and react to some inputs at the same time, while processing means read, iterate over the bits of each read portion (Bytes atm) and write an output pin accordingly, with a precise timing of 4.4 µs each.

So let's change the scale a bit, here's my measuring code:

Code: [Select]
#include <SPI.h>
#include <SD.h>         // modified version

File srcFile;

void setup() {
  if (!SD.begin(4, SPI_FULL_SPEED)) {
    return;
  }
  srcFile = SD.open("eight.three");
  if (srcFile) {
    unsigned long t_start = micros();
    for (int i = 0; i < 5632; i++) {
      byte bla = srcFile.read();
    }
      unsigned long t_end = micros();
      Serial.begin(250000);
      while (!Serial) {
      }
      Serial.print(t_end - t_start);
    srcFile.close();
  }
}

void loop() {
}


The output using a "gnome-disks"-fat32-formatted card is: 186588
Same program using the same card, but formatted by SDFormatter4 under Windows (hard inough to manage that ;p ) outputs: 192384
That's a hell of a lot more (recall the 200,000 µs total).


Now here's my actual question: How, if not using a certain formatting tool, will I get this job done in about 100,000 µs?

I actually do need a filesystem btw. I've already had a lot of thoughts about omitting it - finally that's no option. Going raw is a thing I'll definitely try in some other project, though.

Thanks in advance
  C1913

Arduino Uno, sorry ^^
If google was a country, I'd move there.

fat16lib

#18
Oct 08, 2015, 05:13 am Last Edit: Oct 08, 2015, 05:57 am by fat16lib
C1913,

Your performance problem is not related to format.  Single byte read is very slow and SD access has almost nothing to do with read speed.  I am not sure why you see a slight difference in read speed, single byte read should be almost independent of the SD or format.

What version of SD.h are you using that accepts "eight.three" as a file name?

You need to read multiple byte into a buffer.  You should be able to get a large increase in read speed, at least a factor of ten.

Look at the bench.ino example in SdFat.  It can read at up to 400 KB/sec on an Uno.

Edit:

I ran your example with a correctly formatted SD using SdFat and it printed 76140.  Not sure why you get such a long time for single byte read.

I ran bench.ino with a 64 byte buffer and got the following
Quote
File size 1 MB
Buffer size 64 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
283.37,48164,64,220

Starting read test, please wait.

read speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
320.92,2056,68,193

C1913

First, thanks for the reply.
It's he SD library that comes with Arduino 1.6.5. The modification I made is to have it accept an optional parameter for SPI speed:
boolean SDClass::begin(uint8_t csPin, uint8_t spiSpeed = SPI_HALF_SPEED)

"eight.three" is just a replacement for the actual file name, a well known 8.3 styled name that I didn't want to be recognized here. Sorry for that.

The bench.ino indeed output something just above 300 kByte/s reading, didn't have time to take a closer look at it, though.
I'll switch to SdFat and try some project related things with it asap. Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.


Allthough it no longer belongs here, for
Your performance problem is not related to format.
I'm gonna post the results as soon as I'll get some.
If google was a country, I'd move there.

fat16lib

#20
Oct 09, 2015, 12:40 am Last Edit: Oct 09, 2015, 02:40 am by fat16lib
Quote
Funny thing btw, running that exact same code I posted before, only using SdFat.h instead of SD.h, I lose another 26 ms (218,180 µs total). Something's weird here, but I'm on it.
Yes, makes no sense.  When I run your code with SdFat I get 76140, almost three time faster.  I copied the code and changed SD chip select from 4 to 10 to match my shield.

Edit: I ran the program again and got a longer time.  I will investigate more.

Still, the answer for better performance is to do multiple byte reads.

Here is a buffered read that take 16836 micros.

Code: [Select]

uint8_t buf[64];
    unsigned long t_start = micros();
    
    const size_t TO_READ = 5632;
    size_t nr;
    for (int i = 0; i < TO_READ; i += nr) {
      nr = TO_READ - i;
      if (nr > sizeof(buf)) nr = sizeof(buf);
      size_t ir = srcFile.read(buf, nr);
    //  byte bla = srcFile.read();
      if (ir != nr) {
        // read error
        return;
      }
      for (int j = 0; j < nr; j++) {
        // process bytes here
      }
    }
    unsigned long t_end = micros();

bitNine

Funny, but I was having endless issues getting an SD card to work on an ethernet shield. After following every suggestion available on the internet, I finally broke down and bought a different shield. Got the Arduino Ethernet shield 2, and popped in the card. STILL didn't read. Argh... Ran across this thread, reformatted the card with the SD formatter, and it started working immediately. Thankfully, it still didn't work in the other shield, so at least the new shield was worth the purchase.

What makes me crazy is that there's this page: https://www.arduino.cc/en/Reference/SDCardNotes

It says, "Windows : right click on your card's directory and choose "Format" from the drop down. Make sure you choose FAT as the filesystem."

It's so outdated that it says nothing about FAT32, doesn't say anything about cluster size, and literally says to use the OS formatting. That's what I read long before I found this thread, and it misled me. Someone needs to change it.

Skylark

Does this apply to usb flash memories as well or just SD cards?

fat16lib

This post is for SD cards.

USB drives used on Arduino are more forgiving than SD cards. 

You should format the USB drive with a single MBR based FAT16/FAT32 partition.

Expect poor performance with USB flash drives on Arduino.

NattyFido

I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card! :smiley-confuse:

C1913

There's a license free version of Windows XP for virtual machines you can download for free. For problems like yours I always have a VirtualBox installed with that version.
Might not be the best solution just for formatting SD cards   :D bit at least it is one.
If google was a country, I'd move there.

fat16lib

Quote
I have no choice but to use the OS formatter for my SD cards. There isn't a Linux version of SD Formatter and running it in Wine, it doesn't see the SD card!
You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.

NattyFido

You can download SdFat and use the SdFormatter example on your Arduino.

I designed the SdFormatter example to produce the same layout as the Win/Mac version of the SD Association formatter.
Thanks! That worked and fast too!

pankaj554

 If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.

fat16lib

#29
Dec 10, 2015, 02:16 pm Last Edit: Dec 10, 2015, 02:22 pm by fat16lib
Quote
If you are using windows operating system, then you can format your SD card, very quickly by set NTFS or fat32 file system. But in the Linux system you will have go through command.
Please read before you post.  The format won't be optimal for Arduino if you use OS utilities.  Arduino is like not Windows, OS X, or Linux.  Desktop OSs have huge amounts of RAM for buffering so the format is less important.

I suggest you learn about the internals of SD cards and how FAT16/FAT32 is implemented for Arduino before providing advice.

Go Up