I've been playing around with writing to SD cards, and have noticed that the time taken to write data seems to be a bit variable. I've tried three different SD cards, and most of the time a write() will take round about 3-4ms, but every now and again it may take a lot longer (over 100 ms on some occasions)
Is there a reason for this, and is there anyway of predicting when a slow write() is about to occur?
This is important to me for the following reasons....
99% of the stuff my application does needs to be done periodically, and almost all of it will take a known (and constant) time. So for example I update my LCD screen 4 times a second and I know it will take about 4ms to complete. I process data from a strain gauge about 10 times a second, and each time it takes about 12ms. etc etc
I'm trying to log and process data at regular intervals, since I know how often stuff needs to be processed and how long it's likely to take, I can 'schedule' operations to make sure nothing gets missed.
The problem is that having a single process (SD write operations) that takes an unknown duration can really screw up my timings.
Example - I have a serial device that generates about 200 bytes of data 10 times a second at a rate of 56000 baud. This data arrives as regular as clockwork, but because the baud rate is so fast it will fill up the 64byte buffer on a mega in about 1ms. This means that I need to be ready and waiting for the data - generally not a problem because I know when it's due so I know not to start a process that may take 4ms to complete 1ms before the serial data is due to arrive.
The problem comes with the SD write(), since I don't know whether it's going to take 4ms or 40ms, I don't know if it's going to finish in time to read from the serial device.
All sdcards have got "write latency". The WL ranges from a few ms up to 250ms. It is random. You cannot do much with it. The sdcard does its internal housekeeping from time to time.. Or better to say "all the time"
The best way to avoid data loss is to introduce a FIFO - a buffer in ram, which will consume your periodic writes, and store the data out of FIFO to the sdcard when possible.
Search the forum - it has been discussed many times.. http://forum.arduino.cc/index.php?topic=144715.msg1092515#msg1092515
The WL ranges from a few ms up to 250ms. It is random. You cannot do much with it.
Damn - I feared this might be the case!
introduce a FIFO - a buffer in ram
I've already done this, but all it does is reduce the overall write time for the entire process, it doesn't actually reduce the uncertainty about how long your next write is going to take.
I have a logging application that attempts to log a 50 character message several times a second. By introducing a 500 byte buffer I only need to do a write() once every 10 records, but I still have no way of guessing how long that write might take. This technique also relies on you having 500 bytes of ram available !
I did with NilRtos (read the forum topics, maybe 3y back). There is a topic with many details on it, with pictures. There is a FIFO implementation for NilRtos (from fat16lib).
I was able to store 100bytes large data record (with various data) 333x per second (33kB/sec) into an average sdcard, atmega1284p at 16MHz. The FIFO was 4kB deep I think.
The FIFO size:
FIFO = Max_Write_Latency_in_seconds * Bytes_per_second_to_write
For example 500bytes 10x per second and 100ms max write sdcard latency (an estimate..)
FIFO = 0.1 * (500*10) = 500bytes deep fifo.
Maybe you have rearrange your app / use NilRtos for example, it will work..
For example (with NilRtos):
. Task1 will be a "periodic task" with 10ms period, will gather data from somewhere (a data record) and write the record into the FIFO while the FIFO is not full.
. Task2 will "continuously" read the data records out of the FIFO and try to write them onto the sdcard while FIFO is not empty.
In case the FIFO buffer is large enough, there will not be an FIFO overrun during sdcard's Write Latency hit, and the data stream will be written onto the sdcard without a data loss..
