Using sdfat lib - question about SD Card hardware and sample rate

Hi all,

I'm currently working on a university project.
We want to log data from several sensors (connected by I2C Bus) to a micro SD card using SPI and the sdfat-lib on an arduino Uno (SD Card Reader Catalex from ebay).

In this context I would like to kind of dig out this post:
http://forum.arduino.cc/index.php?topic=49649.0

We used the procedure of "controlling the flush call" as described in the post above to get a high sample rate (50/s would be great, but is not mandatory).
We also used a class 10 card and benchmarked it with the "bench-code" (write speed: 249kb/s, max latency 24451µs, avg latency 2049µs)

Thereby we managed to increase our sample rate to about 20-25/s (already quite good), thanks for that first of all ;)!

We will store about 20 values in a cvs file as float with 5 digits after decimal point (e.g.: 6216.00000;1014.87750;23.04688; .... ). Currently we call a flush every 1000 measurement rows and it works quite good (see code fragment for details)
We use a pushbutton to start/stop the measurement and open/close the sd file on the event. We will log the data for about 5min.

#define DATA_LENGTH 20
#define FLUSH_RATE 1000
//[...]
file.open(FILENAME, O_CREAT | O_WRITE | O_APPEND)	//Open the sd file when button is pushed
//[...]
  // Write Sensor Values
  for (uint8_t i = 0; i < DATA_LENGTH; i++) 
  {
    file.print(DatatoSD[i],5);    // Write the data array to the SD Card Controller(?)
    file.write(';');		//separator after each value
  }
  file.println();  // one row of the data array is ready, so make a new line
  
  if(count%FLUSH_RATE==0)	//count just counts the measurement rows logged
  {
  file.flush();  
  }

We don't really care about data loss in case of e.g. power failure because we can simply perform the measurement again. Hence flushing only every 1000 measurement rows would not be an issue for us.

My questions, since I would like to explain it correctly in my thesis:
1)
It was mentioned that the sdfat-lib has a internal buffer of 512bytes. As we obviously now put more than 512bytes "somewhere" with the print- (and write) method:
Where does the data go? Is it already going to the SD card? What is the max. amout of data we can put "there" before we have to call the flush again not risking an "overwrite" or something like that? Is there official documentation which explains how SD card work on hardware level?

  1. In order to possibly increase the sample rate to about 50/s? Is that possible for a beginner without using very sophisticated coding?

Thanks a lot upfront, sorry for the long post and all the best
Tim

If you don't care about data loss then you don't have to flush at all. Just close the file when you're done collecting data.

The library writes the data to the card whenever the buffer fills. But unless you close the file (or call flush) the FAT doesn't get updated and it will appear as if nothing was written.

For 50 records per second, that's 20ms per sample. Your latency sometimes exceeds that so it would affect the timing. I think you would have to do something a little more sophisticated in order to maintain your sampling rate, or accept the fact that occasionally there is a glitch. Someone else will have better advice on this than me.

We will store about 20 values in a cvs file as float with 5 digits after decimal point
(e.g.: 6216.00000;1014.87750;23.04688; .... ).

Some thoughts...

  1. Given that the Arduino float is an IEE754 32 bit, which has max 7 significant digits why all those decimals?

  2. You could go for writing the floats in binary form. That is only 4 bytes per number instead of 8. Yes you need to unpack these later (can be done on the Arduino, or on PC) Should give you about the factor 2 you are looking for.

  3. I assume these floats come from some measurement & some converting? probably it is faster to log the raw measurements (especially when integer). ALso these raw data can be written in binary form.

To see if the above remarks make sense, please post the complete sketch.

Hi,

thanks for the quick answers!

But unless you close the file (or call flush) the FAT doesn't get updated and it will appear as if nothing was written.[...]For 50 records per second, that's 20ms per sample. Your latency sometimes exceeds that so it would affect the timing

Very good, so we will just close the file at the end I think. But how does that work on hardware level. Why does the updating of the FAT takes so long? What causes the latency? What is meant here by the "timing"?

  1. I assume these floats come from some measurement & some converting? probably it is faster to log the raw measurements (especially when integer). Also these raw data can be written in binary form.

Correct, they are coming from converting from the integer "counts" to pressure or temperature. Loggin of the raw int values is a good idea, thanks. But how do I write data in binary form which may deliver the factor of 2?

Nevertheless we performed some tests today:

We saved a time stamps between opening, reading, writing and flushing in the arduino program and plotted them, see picture. You can see how long the reading/writing for each measurment row has taken. Now we clearly see that the most time consuming is the writing. But what are those strange "quite regular" peaks there? Is this coming from the latency?

Furthermore we tested our class 10 SanDisk SD card again. We formatted the card with the OS formatting tool and with the SD formatter. We also tried different cluster sizes (2kb - 64kb). Sometimes we get quite different results for the latency of the card up the 100ms.

Formatted with SDFormatter:

Manufacturer ID: 0X3
OEM ID: SD
Product: SL08G
Version: 8.0
Serial number: 0X8DC6D428
Manufacturing date: 11/2014

File size 5 MB
Buffer size 512 bytes
Starting write test, please wait.

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
249.73,17916,1672,2044
249.40,23772,1676,2047
246.86,90740,1680,2068
247.11,90588,1676,2066
246.82,90988,1668,2068
246.14,90832,1676,2074
246.73,90952,1676,2069
247.14,90528,1668,2065
246.98,99720,1668,2067
Formatted with OS Formatter and 2048byte:

write speed and latency
speed,max,min,avg
KB/Sec,usec,usec,usec
244.90,25052,1676,2084
240.42,55752,1708,2123
240.18,21672,1728,2125
239.37,17916,1732,2133
238.09,27528,1736,2144
237.66,27340,1740,2148
236.67,26348,1748,2157
242.36,32532,1696,2106
241.43,29132,1704,2115
238.94,24760,1728,2136

Unfortunately we cannot figure out any correlation between cluster size/method of formatting and the performance of the SD card...it look like this happens almost at random. Could this be this be possible? An if how/why?

Thanks again and take care
Tim

McBane:
Why does the updating of the FAT takes so long? What causes the latency? What is meant here by the "timing"?

It requires more read/write cycles to update the FAT. The timing I was referring to was your sample period. If you need to read your sensors every 20ms but every so often it takes 25ms to write to the SD then you'll be late or miss readings, assuming you're doing things linearly. I don't know what you're doing though. It's possible to fix that problem as long as the average latency doesn't exceed the remaining processing time you have. It depends on what you're doing.

But what are those strange "quite regular" peaks there?

I'm not sure. But just eyeballing it, it looks like you get one roughly every 32K bytes written. Maybe that's the cluster size. Just guessing though.

The SD cards contain processors that perform complicated operations (e.g. wear leveling) so their performance from write to write isn't quite predictable. And they differ from card type to card type. At least that's my very limited understanding. Somebody else will explain it better.