Simple data logger sampling problems

Hello everyone,

I am working on a simple data logger that records values from 3-4 analog pins and writes them to an SD card at roughly 100 Hz. When I go to plot 20 seconds worth of data, I only have about 500 samples.

I have looked at examples that use up to 5000 Hz but they are all way to complicated for me to understand.

Is there any way to modify my code that would fix the sample rate problem? For that matter is it even a sampling rate problem?

/*
 * Original code by "felipearcaro" https://forum.sparkfun.com/viewtopic.php?f=32&t=38500
 * Modified to work with FSR sensors by Taylor Rucker, Jan. 2016.
 */

#include <SD.h>

//manually input desired filename here before each test
//input subject initials and the last 2-3 characters of shoe type. omit the first two characters of shoe type as only 8.3 names are supported.
//example: "TR45" "TR70" or "TR45T" etc
char *s = "TR45";

//initialize pins for SD shield
const int chipSelect = 4;

// make it long enough to hold your longest file name, plus a null terminator
char filename[16];

//initialize pins for sensors
int sensor1Pin = 0;
int sensor2Pin = 1;
int sensor3Pin = 2;


void setup()
  {
    //open serial communications and wait for port to open:
    Serial.begin(9600);

  //make sure that the default chip select pin is set to output, even if you don't use it:
  Serial.print("Initializing SD card...");
  //pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");
  int n = 0;  
  snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n); // includes a two-digit sequence number in the file name
  while(SD.exists(filename)) {
    n++;
    snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n);
  }
  File dataFile = SD.open(filename,FILE_READ);
  Serial.print("Trial number: ");
  Serial.println(n);
  Serial.println(filename);
  dataFile.close();
  //now filename[] contains the name of a file that doesn't exist
  }
  
void loop()
  {
    File dataFile = SD.open(filename, FILE_WRITE);
    if (dataFile) {
      float sensorVal[] = {0, 0, 0};
      
      //read sensor values
      sensorVal[0] = analogRead(sensor1Pin);
      sensorVal[1] = analogRead(sensor2Pin);
      sensorVal[2] = analogRead(sensor3Pin);
      
//optional, use for debugging

      //print values to the serial monitor
      Serial.print(sensorVal[0]);
      Serial.print(",");
      Serial.print(sensorVal[1]);
      Serial.print(",");
      Serial.println(sensorVal[2]);

//

      //print values to data file
      dataFile.print(sensorVal[0]);
      dataFile.print(",");
      dataFile.print(sensorVal[1]);
      dataFile.print(",");
      dataFile.println(sensorVal[2]);
      dataFile.close();
    }
    else {
      Serial.println("error");
 }
 //delay in miliseconds (10 = 100 Hz)
 delay(10);
  }

I’m using an Uno, Adafruit SD shield and an 8 Gb micro SD, please let me know if you need more info.

Thanks!

delay(10); won't give a repeat rate of 100 per second because it takes no account of the time required in the rest of the code in loop().

Have a look at how time is managed using millis() in Several Things at a Time

I have only messed about with an SD card once so I am not an expert. I would have thought it was sufficient to open the file once, rather than in every iteration of loop().

It may also be more efficient to take several readings and store them as a group rather than one at a time - that depends on exactly how the SD Card library works.

...R

Robin is correct about the delay involved in all the opening/writing and closing of the files. The reason for opening / closing the file on each pass through the loop (at least typically for logfiles) is so that when the system goes down, you don’t lose the data that was written to the file after the last open but before the file was closed. Typically, you open a logfile, append data and close it each time you add a “block” (whatever that may be) to the file. That makes sure that the data in the file is as current as possible when you go to read it back later either on that device or in another machine for example in the case of a SD card.

Instead of opening/closing the file each loop, open it in setup and don't close it until you signal it to close. You can do that with a button or by entering a character through the serial monitor. Or you could have it close after a certain amount of time or number of loops.

If the code crashes before the file is closed you'll lose your data. If that's a concern you can close/re-open the file in your loop every so often. Those loops will probably take longer than 10ms.

Do that, and replace delay() with code that checks the elapsed time like Robin suggested, and you shouldn't have a problem running at 100Hz.

Thanks for your input everyone.

I stole the opening and closing file code from another program, this was in an effort to increment a number in the file name after every power cycle. The tests I'm running will only be about 15s long at a time so I can't imagine many problems happening in that amount of time. I will do my best to make your suggestions work. I will post the code whenever I need more help or hopefully when it is working.

One other (probably minor) point is that SD card write latency is unpredictable in general because of the internal housekeeping activities of the SD card processor. Most of the time it is relatively short but out of the blue it can be longer, much longer. So you may find on occasion, depending on the specific card, that you have a measurement interval that is a lot longer than 10ms. If that matters to you there are solutions that involve buffering. They're probably the examples you found were too complicated.

Ok guys I’m waving the white flag. I’m stuck.

I have tried following the “Several Things” example and I think getting close but I have an error. If I understand the sketch, the data file should be opened in the logData function and write the data. Then after a set interval, the closeDataFile function should close the file (thus saving data in case the program crashes). However in the closeDataFile function, “close.Datafile();” was not declared in this scope.

Am I understanding this correctly or am I still way off?

#include <SD.h>

// ======Constants

//initialize pin for SD shield
const int chipSelect = 4;

//initialize pins for sensors
int sensor1Pin = 0;
int sensor2Pin = 1;
int sensor3Pin = 2;

//user input. before each trial the filename can be customized here
char *s = "TR45";
char filename[16];

// data file close interval
const int interval = 5000;



// ======Variables

unsigned long currentMillis = 0;    // stores the value of millis() in each iteration of loop()
unsigned long previousSdMillis = 0;   // last time the data file was closed 

void setup() {
   Serial.begin(9600);
   Serial.println("Starting Sketch");

  //make sure that the default chip select pin is set to output, even if you don't use it:
  Serial.print("Initializing SD card...");
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");
  int n = 0;  
  snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n); // includes a two-digit sequence number in the file name
  while(SD.exists(filename)) {
    n++;
    snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n);
  }

  // open the data file
  File dataFile = SD.open(filename,FILE_READ);
  //close data file. Sequence numbering does not work unless file is closed here
  dataFile.close();

   
}

void loop() {

  currentMillis = millis();

logData();
closeDataFile();

}

void logData() {

  File dataFile = SD.open(filename, FILE_WRITE);
    if (dataFile) {
       float sensorVal[] = {0, 0, 0};
       //read sensor values
       sensorVal[0] = analogRead(sensor1Pin);
       sensorVal[1] = analogRead(sensor2Pin);
       sensorVal[2] = analogRead(sensor3Pin);

       //print values to data file
       dataFile.print(sensorVal[0]);
       dataFile.print(",");
       dataFile.print(sensorVal[1]);
       dataFile.print(",");
       dataFile.println(sensorVal[2]);
       }
    else {
      Serial.println("Error");
    }
  }

void closeDataFile() {

  if (currentMillis - previousSdMillis >= interval) {
    dataFile.close();
    previousSdMillis += interval;
  }
}

Thanks for the help everyone, I appreciate it.

I don't think you can repeatedly open a file that's already open. Maybe. You should probably check to see if it's already open first.

You have two variables of type File named datafile in two different functions. Just declare one, globally.

You're also logging at whatever full speed happens to be. I thought you wanted 100Hz?

jboyton,

I see the datafile error, I noticed in my setup it is a file read command and in the functions it is a write command. I re-named them accordingly and made the write variable global. I’m no longer getting that error.

Now when I run the program, no file is created on the SD card.

100 Hz is the minimum rate that I would like, however up to 500Hz wouldn’t hurt at all. I guess I need more help. Do I need a similar if statement in the logData function to specify my sampling rate?

#include <SD.h>

// ======Constants

//initialize pin for SD shield
const int chipSelect = 4;

//initialize pins for sensors
int sensor1Pin = 0;
int sensor2Pin = 1;
int sensor3Pin = 2;

//user input. before each trial the filename can be customized here
char *s = "TR45";
char filename[16];

// data file close interval
const int interval = 5000;



// ======Variables

unsigned long currentMillis = 0;    // stores the value of millis() in each iteration of loop()
unsigned long previousSdMillis = 0;   // last time the data file was closed 

void setup() {
   Serial.begin(9600);
   Serial.println("Starting Sketch");

  //make sure that the default chip select pin is set to output, even if you don't use it:
  Serial.print("Initializing SD card...");
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");
  int n = 0;  
  snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n); // includes a two-digit sequence number in the file name
  while(SD.exists(filename)) {
    n++;
    snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n);
  }

  // open the data file
  File dataFileRead = SD.open(filename,FILE_READ);
  //close data file. Sequence numbering does not work unless file is closed here
  dataFileRead.close();

   
}

void loop() {

  currentMillis = millis();

logData();
closeDataFile();

}

File dataFileWrite = SD.open(filename, FILE_WRITE);

void logData() {
  
    if (dataFileWrite) {
       float sensorVal[] = {0, 0, 0};
       //read sensor values
       sensorVal[0] = analogRead(sensor1Pin);
       sensorVal[1] = analogRead(sensor2Pin);
       sensorVal[2] = analogRead(sensor3Pin);

       //print values to data file
       dataFileWrite.print(sensorVal[0]);
       dataFileWrite.print(",");
       dataFileWrite.print(sensorVal[1]);
       dataFileWrite.print(",");
       dataFileWrite.println(sensorVal[2]);
       }
    else {
      Serial.println("Error");
    }
  }

void closeDataFile() {

  if (currentMillis - previousSdMillis >= interval) {
    dataFileWrite.close();
    previousSdMillis += interval;
  }
}

You haven't addressed either of the last two issues that were raised.

aarg: You haven't addressed either of the last two issues that were raised.

I'm doing my best but I am still very new to programming.

I do not know how to address the 100 Hz issue, I was hoping to get advice in the next replies.

I also thought that I took care of the using dataFile twice issue but I could be wrong.

As for the opening of the file multiple times, I thought the file was opened and closed in the setup, then opened before the logData function, and lastly closed after a specified interval in the closeDataFile function.

Sorry if it takes me a while to grasp concepts. That is why I'm on the forms trying to learn.

Ok everyone,

Good bad or ugly, I managed to get the sketch working. (To my needs anyways)

I’m going to post the sketch just in case someone can use it and of course for further criticizing :stuck_out_tongue:

/*
 * Original SD sequence code by "felipearcaro" https://forum.sparkfun.com/viewtopic.php?f=32&t=38500
 * SD logging code modified from "datalogger" example in SD library.
 * Modified to work with FSR sensors by Taylor Rucker, Mar. 2016.
 * 
 * This sketch will read values from analog pins 0, 1, & 2 and write them to 
 * a csv file on an SD card.
 * 
 * The sample rate will be as fast as possible. The sketch can probably
 * be modified to have a user-set sampling rate, this was not a feature
 * that I needed at the time. 
 * 
 * PS: This is one of my first sketches, give me a break on my formatting!
 */
 
#include <SD.h>

//===== USER INPUT ======

//Before each trial the filename can be customized here.
//There will be a two digit sequencing number that increments
//after each power cycle.
char *s = "TR45";

//This is the amount of time the data will be written to the SD card
//before the program ends. (The SD file will be closed)
const int interval = 5000; //time in milliseconds

// ====== VARIABLES ======

//declare filename as a character string
char filename[16];

//initialize pin for SD shield
const int chipSelect = 4;

//initialize pins for sensors
int sensor1Pin = 0;
int sensor2Pin = 1;
int sensor3Pin = 2;


//Setup array of sensor values
float sensorVal[] = {0, 0, 0};

//declare dataFile as File Type
File dataFile;

void setup() {
   Serial.begin(9600);
   Serial.println("Starting Sketch");

   //make sure that the default chip select pin is set to output, even if you don't use it:
   Serial.print("Initializing SD card...");
  
     // see if the card is present and can be initialized:
     if (!SD.begin(chipSelect)) {
       Serial.println("Card failed, or not present");
        // don't do anything more:
        return;
  }
  
   Serial.println("card initialized.");
   int n = 0;  
   snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n); // includes a two-digit sequence number in the file name
     while(SD.exists(filename)) {
       n++;
       snprintf(filename, sizeof(filename), "%s_%02d.csv", s, n);
    }

   // Read the data file (a part of the sequencing code)
   dataFile = SD.open(filename,FILE_READ);
   dataFile.close();
   //now filename[] contains the name of a file that doesn't exist

   //open the file (preparing to write values)
   dataFile = SD.open(filename,FILE_WRITE);
   
}

void loop() {

  if (dataFile) {
       float sensorVal[] = {0, 0, 0};
       //read sensor values
       sensorVal[0] = analogRead(sensor1Pin);
       sensorVal[1] = analogRead(sensor2Pin);
       sensorVal[2] = analogRead(sensor3Pin);

       //print values to data file
       dataFile.print(sensorVal[0]);
       dataFile.print(",");
       dataFile.print(sensorVal[1]);
       dataFile.print(",");
       dataFile.println(sensorVal[2]);

       //when the set interval has passed, close the data file
       //and stop the program (endless loop) 
       if (millis() >= interval) {
     
     //==== OPTIONAL FOR DEBUGGING ====
        /*
        Serial.println("STOP");
        Serial.print("Time Elapsed:");
        Serial.println(millis());
        */
        dataFile.close();
        while(1);
        
       }
          }
//==== OPTIONAL FOR DEBUGGING ====
   /*       
   else {
   Serial.println("error");
    }
    */
}

Thanks again for the help,

Taylor