Improving SD card logger script

Hello!
I have a simple SD card logging script that reads temperature value from LM-35 sensor and writes it to SD card.

#include <SD.h>
File file;
const int pin = 0; 
int tempc = 0;

void setup() {
pinMode(13, OUTPUT);
Serial.begin(9600);
}

void loop() {
file = SD.open("TEST_SD.TXT", FILE_WRITE);
for(int i = 0; i < 8; i++) {tempc += analogRead(pin);}
tempc = (5.0 * tempc * 100.0) / (8.0 * 1024.0);
file.println(tempc,DEC);
file.close();
}

How can i do this without the "FILE CLOSE" in loop?
The problem is that when i add a clock module to my project, it stops logging when the log-file is about 100 kb (although the memory card is 2 Gb).
I think it is because Arduino Duemilanove saves something into its memory (RAM ?) and it just gets full... also it might be because of the clock module - arduino keeps the time values in memory?

The code for SD + CLOCK + TEMP:

#include <SD.h>
#include "Wire.h"
#define DS1307_I2C_ADDRESS 0x68
File file;
const int pin = 0;
int tempc = 0;
byte decToBcd(byte val)
{
  return ( (val/10*16) + (val%10) );
}
byte bcdToDec(byte val)
{
  return ( (val/16*10) + (val%16) );
}
void setDateDs1307(byte second,        // 0-59
                   byte minute,        // 0-59
                   byte hour,          // 1-23
                   byte dayOfWeek,     // 1-7
                   byte dayOfMonth,    // 1-28/29/30/31
                   byte month,         // 1-12
                   byte year)          // 0-99
{
   Wire.beginTransmission(DS1307_I2C_ADDRESS);
   Wire.send(0);
   Wire.send(decToBcd(second));  
   Wire.send(decToBcd(minute));
   Wire.send(decToBcd(hour));  
   Wire.send(decToBcd(dayOfWeek));
   Wire.send(decToBcd(dayOfMonth));
   Wire.send(decToBcd(month));
   Wire.send(decToBcd(year));
   Wire.endTransmission();
}
void getDateDs1307(byte *second,
          byte *minute,
          byte *hour,
          byte *dayOfWeek,
          byte *dayOfMonth,
          byte *month,
          byte *year)
{
  Wire.beginTransmission(DS1307_I2C_ADDRESS);
  Wire.send(0);
  Wire.endTransmission();

  Wire.requestFrom(DS1307_I2C_ADDRESS, 7);
  *second     = bcdToDec(Wire.receive() & 0x7f);
  *minute     = bcdToDec(Wire.receive());
  *hour       = bcdToDec(Wire.receive() & 0x3f);
  *dayOfWeek  = bcdToDec(Wire.receive());
  *dayOfMonth = bcdToDec(Wire.receive());
  *month      = bcdToDec(Wire.receive());
  *year       = bcdToDec(Wire.receive());
}

void setup() {
 pinMode(13, OUTPUT);
byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;
Wire.begin();
  Serial.begin(9600);

  second = 00;
  minute = 39;
  hour = 00;
  dayOfWeek = 6;
  dayOfMonth = 19;
  month = 11;
  year = 11;

 if (!SD.begin()) {
   Serial.println("begin failed");
   return;
 }}

void loop() {
  byte second, minute, hour, dayOfWeek, dayOfMonth, month, year;

  getDateDs1307(&second, &minute, &hour, &dayOfWeek, &dayOfMonth, &month, &year);

file = SD.open("TEST_SD.TXT", FILE_WRITE);

for(int i = 0; i < 8; i++) {tempc += analogRead(pin);}
tempc = (5.0 * tempc * 100.0) / (8.0 * 1024.0);

file.print(tempc,DEC);
file.print(" deg C "); 
  file.print(hour, DEC);
  file.print(":");
  file.print(minute, DEC);
  file.print(":");
  file.print(second, DEC);
  file.print(" ");
  file.print(dayOfMonth, DEC);
  file.print("/");
  file.print(month, DEC);
  file.print("/");
  file.print(year, DEC);
  file.print(" Day_of_week:");
  file.println(dayOfWeek, DEC);

file.close();

}

The problem is that when i add a clock module to my project, it stops logging when the log-file is about 100 kb (although the memory card is 2 Gb).

Does it stop logging, or does it simply take longer to log the data?

Why not create a new file before the old one gets that big? One file per day, or week, or month, or whatever.

Is the temperature really changing that much? Logging once every 5 minutes is probably good enough, isn't it?

Your files may be just getting too cumbersome. Since you have an on board clock, you can use it to name a series of files. This may fix your problem simply by having smaller files. Even if it doesn't, you may ultimately find it a better way to manage files.

You might find this code useful. I simply call it up once at midnight

#include <SD.h>
#include "RTClib.h"
#include <Wire.h>
#include <string.h>


RTC_DS1307 RTC;

char filename[] = "00000000.CSV";

File myFile;


void setup()

{

Serial.begin(9600);

Wire.begin(); //Important for RTClib.h

RTC.begin();

if (! RTC.isrunning()) {

Serial.println("RTC is NOT running!");

// following line sets the RTC to the date & time this sketch was compiled

// RTC.adjust(DateTime(__DATE__, __TIME__));

return;

}

Serial.print("Initializing SD card...");

// On the Ethernet Shield, CS is pin 4. It's set as an output by default.

// Note that even if it's not used as the CS pin, the hardware SS pin

// (10 on most Arduino boards, 53 on the Mega) must be left as an output

// or the SD library functions will not work.

pinMode(10, OUTPUT);


if (!SD.begin(4)) {

Serial.println("initialization failed!");

return;

}

Serial.println("initialization done.");


}

void loop()
{

getFileName();

createFileName();

delay(3000);

}

void getFileName(){

DateTime now = RTC.now();

filename[0] = (now.year()/1000)%10 + '0'; //To get 1st digit from year()
filename[1] = (now.year()/100)%10 + '0'; //To get 2nd digit from year()
filename[2] = (now.year()/10)%10 + '0'; //To get 3rd digit from year()
filename[3] = now.year()%10 + '0'; //To get 4th digit from year()
filename[4] = now.month()/10 + '0'; //To get 1st digit from month()
filename[5] = now.month()%10 + '0'; //To get 2nd digit from month()
filename[6] = now.day()/10 + '0'; //To get 1st digit from day()
filename[7] = now.day()%10 + '0'; //To get 2nd digit from day()
//filename[8] = filename + ".csv";
}

void createFileName(){
myFile = SD.open(filename, FILE_WRITE);
myFile.close();
}

PaulS:

The problem is that when i add a clock module to my project, it stops logging when the log-file is about 100 kb (although the memory card is 2 Gb).

Does it stop logging, or does it simply take longer to log the data?

Why not create a new file before the old one gets that big? One file per day, or week, or month, or whatever.

Is the temperature really changing that much? Logging once every 5 minutes is probably good enough, isn't it?

Thank you both for reply..
It simply stops logging. I connected LED to indicate the pauses between recordings. LED works SD-card recoring doesnt.
Last time i started the logger in the evening and in the morning i noticed that it had stopped 3.00 AM.
Logging frequency was about 1 recording in 5 min.

Logging frequency was about 1 recording in 5 min.

That's 12 events per hour.

Last time i started the logger in the evening and in the morning i noticed that it had stopped 3.00 AM.

That's at most a 9 hour stretch.

9 * 12 = 108

The problem is that when i add a clock module to my project, it stops logging when the log-file is about 100 kb (although the memory card is 2 Gb).

This implies that you are logging about 1000 bytes per event.

Something doesn't add up here.

I found my earlier post: http://arduino.cc/forum/index.php/topic,79551.0.html#msg682829
Mr. fat16lib shares some good information:

I am amazed your setup works at all. You need to clean up your wiring so the SD is close to the CPU and use very short wires for SPI.

The version of SD.h included with 0022 opens files with the O_SYNC flag so there is an incredible amount of I/O to the card.

Also opening and closing the file for each data point causes more I/O as the file grows since open does a seek to the end-of-file.

I/O to the card is done in 512 byte blocks and a sync call requires at least 2048 byte of I/O. I suspect you are doing over 50,000 bytes of I/O to the SD for each data point.

Now I must puzzle out the meaning of:
I suspect you are doing over 50,000 bytes of I/O to the SD for each data point.

Now I must puzzle out the meaning of:
I suspect you are doing over 50,000 bytes of I/O to the SD for each data point.

Not all 50,000 of those bytes are written to the file. In fact, a huge majority are not.

There is communication between the Arduino and the SD card that is in addition to that needed to send the data that you want written to the card.

I see a sketch that writes data every time through loop.

I have a project that writes data about 1000 times a day. It goes for weeks without any hiccups. Makes me wonder what's up with yours.

Although I'm not sure it would matter I'd declare tempc as a float instead of int.

Also

for(int i = 0; i < 8; i++) {tempc += analogRead(pin);}

might be problematic as you aren't clearing tempc first. It's probably not stopping your sketch but may be effecting your results. Put a tempc = 0 statement right before the if.

I do use a different library for my 1307 clock though and I wonder if your problem is there. I use the Time.h lib from the Arduino site. It has a lib for the 1307 included. I had no problems with it. I even get very accurate time. It's drifted a few seconds over a few months.

http://www.arduino.cc/playground/Code/time