SD write

I'm trying to make a simple data logger with an Uno as my first arduino project. I just want to write the values of a sensor to a SD card 10 times per second. The problem is that the time it takes to write to the SD card seems impossible to determinate. I created 3 experiments: First, I use an interruption (using timerOne) to set a flag. On the main loop, when that flag is on I write the current value of millis() through the serial port. The graph shows the difference between each consecutive value written: It seems to work! But now on the second experiment, I repeat the same thing except I use the SD library to write the value of micros() to a file. The results differ greatly: On a third experiment, I just write on the main loop 100 values of micros() to the SD card, no interruptions involved, and the variance is even bigger:

Inspecting the source of the SD libraries, I see that when Sd2Card::writeData() is called and there's no hardware SPI (that's the case of Uno, isn't it?) interruptions are blocked, but does that explain the great variance of the results?? and more importantly: is there a solution?

The graph shows the difference between each consecutive value written:

It does, huh?

It seems to work! But now on the second experiment, I repeat the same thing except I use the SD library to write the value of micros() to a file. The results differ greatly:

Can’t see your graph OR your code.

On a third experiment, I just write on the main loop 100 values of micros() to the SD card, no interruptions involved, and the variance is even bigger:

Ditto.

Inspecting the source of the SD libraries, I see that when Sd2Card::writeData() is called and there’s no hardware SPI

Yes, there is.

and more importantly: is there a solution?

Yes. Change line 42.

I double-checked that images are visible, and the HTML of the page looks OK to me (the image tags exist and are well formatted), however if anyone has a problem with their browser not allowing them to see the images, please download them here http://dl.dropbox.com/u/17401485/graphs.zip. Sorry for the inconvenience.

As for the code, here it is:
Experiments 1 & 2:

Experiment 3:

Inspecting the source of the SD libraries, I see that when Sd2Card::writeData() is called when there’s no hardware SPI

Fixed that. So if Uno has hardware SPI and interruptions are not being blocked, what else can be causing the variance?
Thanks for your time

My guess is that the library uses buffering to the block size of the SD card. This means that most writes will just put data into a buffer, and once the buffer has filled up, the library will actually flush the data to the SD card, and then start over with another empty buffer. That flushing can quite possibly take time. Other things that can take time include allocating clusters for the FAT file as it grows in size -- this requires more writing only sometimes.

I double-checked that images are visible

I'm reading this behind a firewall and proxy server that bans images and code from some locations, such as where your pictures are hosted.

Opening and closing the file each time you write to it is killing you. You need to open the file once, write all the data to it, and then close the file.

The downside of this is that the file can become corrupt if the Arduino crashes or the card is removed without closing the file.

You can mitigate the issue by buffering data, and only opening and closing the file when the buffer is (nearly) full.

If you use a circular buffer, and write to the buffer in the interrupt handler, and have loop write the data to the file as the buffer fills up, you can be writing to the buffer and the SD card at the same time.

I rewrote the code following using a FIFO queue:

#include <TimerOne.h>
#include <SD.h>
#include <QueueArray.h>

#define EXPERIMENT "data1.csv"

void setup()
{
  Serial.begin(9600);
  
  Serial.println("Press a key to start");
  while(!Serial.available()) {};
  
  Serial.print("Initializing SD card...");
  pinMode(10, OUTPUT);
   
  if (!SD.begin(10)) {
    Serial.println("initialization failed!");
    return;
  }
  
  Serial.println("Starting experiment.");
  
  Timer1.initialize();
  Timer1.attachInterrupt(set_flag, 100000);
  
}

File myFile;
QueueArray <unsigned long> queue;

void loop()
{
  if(!queue.isEmpty() ) {
    if(!myFile) {
       myFile = SD.open(EXPERIMENT, FILE_WRITE); 
    }
    myFile.println(queue.pop());
    Serial.println(queue.count());
  } else {
    if(myFile) {
      myFile.close();
    }
  }
}

void set_flag(void){
  queue.push(millis());
}

That solved the variance problen. However, the queue keeps growing endlessly:

Maybe 10Hz is already too much?

Maybe 10Hz is already too much?

I wouldn't think so. Try a lower rate, though, and see what happens.

ludovic: ``` ...     myFile.println(queue.pop()); ...

void set_flag(void){   queue.push(millis()); }

Is the QueueArray object interrupt safe? Does it save interrupts inside the push() and pop() functions, and then restore them before returning? Else you may very well have a race between the push updating the queue and the pop updating the queue.