SD card latency help?

Other Hardware Storage
1

Hello!

I have a data logging project where I am logging data from an MPU6050. It logs the data every 5 milliseconds. It is quite good and I have even implemented a real time clock. The problem is that it takes twoce as long every 15th reading. Like the following:

Time between logs
928160 -1284 -1828 11188 -86 -40 -168 5004
933164 -1184 -1784 10988 -78 -51 -172 5004
938168 -1224 -1788 11044 -74 -37 -128 5004
943172 -1376 -1772 11012 -83 -52 -155 5004
948176 -1256 -1720 11024 -67 -18 -143 5004
953180 -1264 -1792 11136 -78 -68 -138 5008
958188 -1320 -1752 11044 -80 -46 -156 5004
963192 -1368 -1748 11068 -85 -44 -154 5008
968200 -1224 -1656 11096 -69 -20 -147 5000
973200 -1304 -1768 11072 -71 -59 -154 5004
978204 -1228 -1760 10968 -104 -34 -140 11392
989596 -1160 -1680 11148 -86 -21 -169 5004
994600 -1200 -1748 11108 -77 -35 -149 5000
999600 -1320 -1672 11096 -76 -42 -137 5004
1004604 -1328 -1712 11140 -88 -39 -162 5004
1009608 -1288 -1792 11068 -70 -37 -168 5004
1014612 -1232 -1732 11036 -83 -19 -141 5008
1019620 -1264 -1644 11064 -98 -46 -139 5004
1024624 -1332 -1764 11008 -83 -43 -138 5004
1029628 -1236 -1708 10964 -88 -36 -146 5008
1034636 -1264 -1728 11108 -94 -48 -133 5008
1039644 -1348 -1780 11076 -56 -37 -137 5004

I am logging with the following code:

Serial.println(micros());
accelgyro.getMotion6(&ax, &ay, &az, &gx, &gy, &gz);

dataFile.print(micros());
dataFile.print(",");
dataFile.print(ax);
dataFile.print(",");
dataFile.print(ay);
dataFile.print(",");
dataFile.print(az);
dataFile.print(",");
dataFile.print(gx);
dataFile.print(",");
dataFile.print(gy);
dataFile.print(",");
dataFile.println(gz);

Serial.println(micros());

I guess my question is: Any advice on how to make the arduino not drop these values. I have tried SDfat and sd and there is no difference. I guess the latency comes form the string conversion. Any ideas?

2

The problem is that it takes twoce as long every 15th reading.

Not surprising. The earlier writes write to a buffer. The 15th one fills the buffer and requires actually writing to the card, which takes some time.

I guess the latency comes form the string conversion.

No, it doesn't, or all the writes would be the same speed.

Any ideas?

Don't log so often. Then, the commit part will complete before you need to write again.

3

Thanks PaulS...

I built a string and then wrote it all at once. It actually made it worse. Maybe there is a better way.....See code below..

      accelgyro.getMotion6(&ax, &ay, &az, &gx, &gy, &gz);
          asis=String(ax)+","+String(ay)+","+String(az)+","+String(gx)+","+String(gy)+","+String(gz)+","+(micros())+","+("\r\n");
          y++;
      
      if (y=30)
      {
dataFile.print(asis);
y=0;
asis="";
    }
    Serial.print(asis);
4

Write latency is inherent to SD cards. The SD spec allows up to 200 milliseconds latency for a write. Occasional write latencies of 100 milliseconds are not uncommon.

You must design your logger to allow for latency. I have posted several example fast loggers.

The cleanest way is to use a RTOS with two threads and read sensors in a high priority thread and buffer the data to be written to the SD in a lower priority thread.

I posted RTOS examples as ChibiOS20120529.zip and FreeRTOS20111031.zip here http://code.google.com/p/beta-lib/downloads/list.

Another way is to read the sensor in an ISR and buffer the data for loop. AnalogIsrLogger20120810.zip posted in the above location is a very fast logger that reads the Arduino ADC in an ISR and write the the data in loop().

Several other demo sketches are in fastLoggerBeta20110802.zip. This file is now a bit old.

5

http://elm-chan.org/docs/mmc/mmc_e.html

Find 'Improving Write Performance' and see the diagram showing multiple sector write vs single sector write. Each time a block of NAND flash is to be written to, it first gets erased and then you write 1 or many sectors before the next erase. You get 2:1, at least your program is being efficient.

The size of the card can make a difference. Small cards, like 128M have smaller blocks to erase.

6

Chan's website is very good but most of the stuff about performance of SD cards is out of date.

Wear leveling in modern TLC flash can dominate latency. Modern cards move data that has not been changed for a long time to a new physical location. Flash controllers have various amounts of RAM buffering and use different algorithms so generalizations like small cards are faster just isn't true.

Write latency also depends on the access pattern. 328 Arduinos have limited memory so there are not many caching options.

Small cards are often formatted with small clusters which increases the overhead since the FAT must be accessed often.

For embedded applications cards with SLC flash tends to be best. Modern consumer cards use MLC or TLC flash. Industrial SD cards mostly use SLC flash. I have had good luck with this cheap Industrial card http://www.newegg.com/Product/Product.aspx?Item=9SIA12K0CT6829.

No card has uniform low write latency. All cards have occasional longer latencies. You must design fast data loggers with this in mind to avoid data loss.

I am now working on improved performance for SdFat with Cortex chips and the Mega. Here is a link to some results for the Due with DMA SPI http://arduino.cc/forum/index.php/topic,134512.0.html.