Go Down

Topic: [Solved] Yet another data logging problem... (Read 2 times) previous topic - next topic

plusminus_

Apr 08, 2012, 01:02 am Last Edit: Apr 17, 2012, 06:38 pm by plusminus_ Reason: 1
Good evening.

I would like some guidance on logging data (writing it to a microSD card in the Ethernet Shield). I want to write data 100 times a second, so quite a lot faster than I could manage with a sketch I wrote today (attached, for reference). Each data point will be 14 bytes long, consisting of one long number (millis()), and 2 floats, separated by commas.

I have read several topics about writing to SD faster than 5 times a second. I read that the SDFat library was faster than the standard SD library, so I have downloaded the December version of SDFat. I am considering using a buffer after reading thread, but I don't know how I will write to buffer and then to the card. And I have read things about using 2 or even 3 buffers, and I am not sure if I will need that.

I will be using a Mega, so I should have enough RAM for a large array, but how big would it need to be? I read here that it was best to write in 512 bytes, so would this mean a buffer of exactly 512 bytes?

Thanks,
+-

robtillaart

#1
Apr 08, 2012, 04:50 pm Last Edit: Apr 08, 2012, 10:08 pm by robtillaart Reason: 1
Best performance is indeed made with buffers equal to the sector size of the SD card (512 bytes).

You say one record consist of 14 bytes, [millis, f1,f2] comma separated fields. But how do you separate records?

Consoder writing pure binary: (the commas do not add anything except 2/14 slack ~14%.
4 bytes long millis
4 bytes float f1
4 bytes float f2

So every 12 bytes is one record. 512 bytes can hold 512/12 = 42 records, leaving 8 bytes .

So in your main loop() you should have a counter that writes the 512 bytes buffer to SD after 42 records.
while writing this buffer you should fill buffer 2. etc

If you think reading 12 bytes records is difficult you could consider writing 16 bytes records of which exactly 32 fit in 512 bytes.
Room for an additional sensor ;)
Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

fat16lib

To log 1000 samples per second requires a totally different technique than your sketch.

I suggest you look at example sketches in fastLoggerBeta20110802.zip here http://code.google.com/p/beta-lib/downloads/list.

Forget advice like 512 byte buffers unless you are doing raw writes to the SD.

The big problem is that a write to a file on an SD can take 200 milliseconds.  You must capture data while the SD write is in progress.  This means interrupts, probably using timers, and cleaver buffering schemes.

robtillaart

I thought that using 512 bytes buffers was allways a good idea, raw and cooked, as low level it will always be written raw ?

A question arose while thinking about your project:
If you do a 1000 samples per second, why add the millis() timestamp?
If evenly divided you just get consecutive numbers, if not you get multiple samples with the same timestamp (implying a zero duration between). using the micros as timestamp gives at least an - sub milli -  interval between two samples.




Rob Tillaart

Nederlandse sectie - http://arduino.cc/forum/index.php/board,77.0.html -
(Please do not PM for private consultancy)

fat16lib

For raw writes you must write 512 bytes.

For file writes there is some gain if all writes are 512 bytes and aligned on 512 byte boundaries.

For most apps this gain is not worth the effort.  My benchmark example in SdFat uses a 100 byte buffer. 

With a SanDisk 1 GB Ultra II card formatted FAT16 with the SdFormatter example I get this result:
Quote

Free RAM: 1029
Type is FAT16
File size 5MB
Starting write test.  Please wait up to a minute
Write 198.11 KB/sec
Maximum latency: 81396 usec, Avg Latency: 500 usec

Starting read test.  Please wait up to a minute
Read 299.33 KB/sec
Maximum latency: 2720 usec, Avg Latency: 329 usec


This is a very good card for data logging.  Not because of the average write rate of 198 KB/sec but the max latency of 81396 usec.  Some cards have a max write latency of 200000 usec.

The 198 KB/sec is way faster than needed to log 1000 records per second with 14 byte records.

Once again the design problem is to overcome the occasional long write latency that is inherent in SD cards.

Even class 10 cards have this problem.  The assumption is that devices like video cameras have lots of buffer so they achieve high average write rates for very large files.  This allows occasional pauses by the card to erase large areas of flash.  You can only write to erased flash.

Go Up