Must have more files! (Datalogging to SD card)

Using Arduino Programming Questions
1

Okay so I'll start off by saying I'm using a Arduino Uno with an Adafruit Datalogger Shield. Right now it makes one file called datalog.txt and it writes to it. Well the problem is, I need it to make a new file for every series of data. So it will turn on, record the data to a .txt file and then be turned of when the test is over. Then when it is booted back up again it needs to make a new file. Here is my code for the SD card in it's current state (What would be really nice is if someone could tell me a simple edit to my current code so I could fix this as painless as possible without having to completely rewrite my code) :

//Calling upon file for the clock
#include <Wire.h>
#include "RTClib.h"

RTC_DS1307 rtc;

File dataFile;

void setup(void) {
   
    Serial.begin (9600);
    
     while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }


  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    while (1) ;
  }
  Serial.println("card initialized.");
  
  // Open up the file we're going to log to!
  dataFile = SD.open("datalog.txt", FILE_WRITE);
  if (! dataFile) {
    Serial.println("error opening datalog.txt");
    // Wait forever since we cant write data
    while (1) ;
  }
   
    pinMode(7,INPUT); // This is the analog comparator negative input.
    // This is where the input signal enters the Arduino.

    SREG = SREG | B10000000; // Enable gobal interrupts. They should 
    // already be enabled but I like to do this out of good measure.

    //Serial.begin(9600); // For printing the frequency to the terminal
    SD.begin();
    
    #ifdef AVR
      Wire.begin();
    #else
      Wire1.begin(); // Shield I2C pins connect to alt I2C bus on Arduino Due
    #endif
      rtc.begin();

    if (! rtc.isrunning()) 
    {
      Serial.println("RTC is NOT running!");
    }   
}
1 Like
2

Replace this code with something that creates a new unique file each time is runs:

dataFile = SD.open("datalog.txt", FILE_WRITE);

For example you could do that by including a number in the file name, and keep incrementing the number until you find a name which doesn't match an existing file.

3

I could but my problem is that I don't know how to code that

4

bhay:
I could but my problem is that I don't know how to code that

Which part are you having trouble with?

Break the problem down into the simplest possible parts. I'm sure you can tackle some of them yourself.

5

Well after reading a book for about 3 hours I done tackled all of it myself but if you wanna check it for problems, that'd be pretty cool.

File dataFile;
File root;
File nextFile;
boolean eof = false;
int count=0;
String file = "";
String fileNum = "";

void setup(void) {
   
    Serial.begin (9600);
    
     while (!Serial) {
    ; // wait for serial port to connect. Needed for Leonardo only
  }


  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    while (1) ;
  }
  Serial.println("card initialized.");
  /*
  // Open up the file we're going to log to!
  dataFile = SD.open("datalog.txt", FILE_WRITE);
  if (! dataFile) {
    Serial.println("error opening datalog.txt");
    // Wait forever since we cant write data
    while (1) ;
  }
   */
   root = SD.open("/");
   while(!eof)
   {
      count++;
     nextFile=root.openNextFile();
     
     if(!nextFile)
     {
       eof=true;
     }
     
     if(eof)
     {
     fileNum = String(count);
     file = "datalog"+fileNum+".txt";
     }
     
     nextFile.close();
   }
   root.close();
   char fileName[file.length()+1];
   file.toCharArray(fileName, sizeof(fileName));
   dataFile = SD.open(fileName, FILE_WRITE);
    pinMode(7,INPUT); // This is the analog comparator negative input.
    // This is where the input signal enters the Arduino.

    SREG = SREG | B10000000; // Enable gobal interrupts. They should 
    // already be enabled but I like to do this out of good measure.

    //Serial.begin(9600); // For printing the frequency to the terminal
    SD.begin();
    
    #ifdef AVR
      Wire.begin();
    #else
      Wire1.begin(); // Shield I2C pins connect to alt I2C bus on Arduino Due
    #endif
      rtc.begin();

    if (! rtc.isrunning()) 
    {
      Serial.println("RTC is NOT running!");
    }
}
6

Here is how I did something similar a while ago. This technique will give you up to 100 incrementally numbered files (00 through 99).

#include <SPI.h>
#include <SD.h>

char fileName[] = "LOGGER00.CSV";  // Base filename for logging.

void setup()
{
  // Construct the filename to be the next incrementally indexed filename in the set [00-99].
  for (byte i = 1; i <= 99; i++)
  {
    // check before modifying target filename.
    if (SD.exists(fileName))
    {
      // the filename exists so increment the 2 digit filename index.
      fileName[6] = i/10 + '0';
      fileName[7] = i%10 + '0';
    } else {
      break;  // the filename doesn't exist so break out of the for loop.
    }
  }
}

I would suggest that every time you need to write you should open the file and then close it immediately after writing, or open the file once and always send a file.flush() after writing. This should help reduce the chance of loosing data, files, or the SD card when you power off the Arduino. I suppose if you really wanted to be safe, have a switch that you throw that disables writing so you can be sure that the write buffers are flushed and you aren't actively accessing the SD card before you intentionally power off. But, that wouldn't cover accidental power outages.

1 Like
7

bhay:

 SREG = SREG | B10000000; // Enable gobal interrupts. They should

// already be enabled but I like to do this out of good measure.

A bit unrelated, but still good for you to know. That line of code is equivalent to:

sei();

EDIT:
With regards to naming the files, the simplest way would be to store a value in the EEPROM of the number of the last file used. Then next time a file is created, simply name it with that number+1 (and store the number to the EEPROM). That way you always know which filename to use, even if files are removed from the SD card.

8

You're using the number of existing files to choose a name for the new file. It might work, but only when all the previous files are present. If you ever deleted one of the historical files, while leaving the most file in place, this algorithm would leave you trying to use the name of an existing file, so it would fail.

The approach Sembazuru demonstrated is closer to what I was trying to describe.

9

PeterH:
You're using the number of existing files to choose a name for the new file. It might work, but only when all the previous files are present. If you ever deleted one of the historical files, while leaving the most file in place, this algorithm would leave you trying to use the name of an existing file, so it would fail.

The approach Sembazuru demonstrated is closer to what I was trying to describe.

Well, with my technique (especially since the Arduino doesn't naturally have an RTC for date stamping files), if one deletes a file from the middle of multiple files, then my technique will fill in the empty spots, essentially destroying any sort of link of chronological order to sequential order. I just needed something quick and dirty when I wrote that.

10

bhay:
, I need it to make a new file for every series of data. So it will turn on, record the data to a .txt file and then be turned of when the test is over. Then when it is booted back up again it needs to make a new file.

Surely, you already use an RTC. This will give you a new file by datestamp. It surely can't get any simpler than this. I just use it daily but you can move or expand to H:M:S as you need, so long as you keep to the 8.3 format - no long filenames allowed.

#include "RTClib.h"              //Date As Filename
#include <string.h>              //Date As Filename 

RTC_DS1307 RTC;
char filename[] = "00000000.CSV"; // this also determines the length of fname
File myFile;

void setup() {
  
         getFileName();
         

void getFileName(){

DateTime now = RTC.now();

   // Turns file_today_name into date.csv format
   sprintf(filename, "%02d%02d%02d.csv", now.year(), now.month(), now.day());
}
11

Hi!

I'm using this routine to create a new log file every month and for time stamping every row of comma separated data inside it:

void writeTempLog()
{
char fileName[13],
_a_dateStr[17];

snprintf( fileName, sizeof(fileName), "TC%02d%04d.LOG", RTCnow.month(), RTCnow.year());
ofstream logFile( fileName, ios::out | ios::app);

snprintf( _a_dateStr, sizeof(_a_dateStr), "%02d.%02d.%04d,%02d:%02d", RTCnow.day(), RTCnow.month(), RTCnow.year(), RTCnow.hour(), RTCnow.minute());
logFile << _a_dateStr << ","
<< ems_Tout << ","
<< ems_Tin_set << ","
<< ems_Tin_FWx00_cur << ","
<< ems_T1_set << ","
<< ems_T1_cur << endl;

logFile.close();
}

12

The scheme to create numbered files will work, but it might take a long time to check for all the numbered files which already exist.

What I did was created a file with the lastest file number in it. The program opens this file, reads the next number, increments it in the file, and then closes the file, and opens the actual data file with the number.

I've found a big problem with sd files becoming corrupted. I'd endorse the suggestion to have a switch which disables file writing before powering the device off.

13

Nick_Pyner:

bhay:
, I need it to make a new file for every series of data. So it will turn on, record the data to a .txt file and then be turned of when the test is over. Then when it is booted back up again it needs to make a new file.

Surely, you already use an RTC. This will give you a new file by datestamp. It surely can't get any simpler than this. I just use it daily but you can move or expand to H:M:S as you need.

Silly me, I didn't take note that bhay (OP) was already using an RTC. Another way of using the RTC to generate file names that are chronologically sequential is using either secondstime() or unixtime(). From RTClib.h:

    // 32-bit times as seconds since 1/1/2000
    long secondstime() const;
    // 32-bit times as seconds since 1/1/1970
    uint32_t unixtime(void) const;

(Hmmm... there seems to be a potential glitch with secondstime() being a signed value...)

A 32bit value represented in HEX will fit an 8 character filename, but isn't quite human readable so I'd also put a human readable date/time stamp at the beginning of each record/line of the file. But it should alphabetically sort properly.

Just change the sprintf(filename, "%02d%02d%02d.csv", now.year(), now.month(), now.day()); line in your sample code to sprintf(filename, "%08x.csv", now.unixtime()); or sprintf(filename, "%08x.csv", now.secondstime());

Though, I suppose (since the SD library supports sub directories) one could use folders with dates as names with files with times as names to have it human readable at the filesystem level.

michinyon:
The scheme to create numbered files will work, but it might take a long time to check for all the numbered files which already exist.

What I did was created a file with the lastest file number in it. The program opens this file, reads the next number, increments it in the file, and then closes the file, and opens the actual data file with the number.

Clever solution. Though I'm not really sure how long the code would take to realize that ...99.CSV is the next available file. And I don't have any of my Arduinos handy to empirically find out.

michinyon:
I've found a big problem with sd files becoming corrupted. I'd endorse the suggestion to have a switch which disables file writing before powering the device off.

One trick that I've developed for at least the prototype stage on my UNOs: If available use two adjacent I/O pins. Configure one pin as an output at LOW, and the other pin as INPUT_PULLUP. I have a 2-pin section of pinheader that I've soldered a wire across the short ends that I use as a male shorting block. No, I wouldn't suggest using this technique as an input switch (bounce would be horrible I imagine), but as a rarely-changed jumper setting it is sufficient and doesn't require any wires to a breadboard.