SD Data logging 86400 seconds is not equal to 24 hours

Hello,

I am running into an issue with my data logging. I am using an UNO to log data onto an SD card (no problems there). I use the millis() function instead of the delay function to make sure I am recording data onto my SD card every 1 second. My recorded data shows that is does in fact record every second.

However, the issue is 86400 seconds does not equal 24 hours. I have kept track of when I start the program and when I end it. So, for example I have uploaded my program at 7:00 am on Aug 10th, and ended the program at 9:00 am on Aug 12th. That results in exactly 50 hours of running time. At 1 second intervals that should be 50 x 60 x 60=180,000 seconds. When I imported my data to Excel I expected to have 180,000 lines of data. To my surprise that was not the case. I ran this experiment multiple times, but always ended up with less time (or Excel rows) than I expected. For instance in one of my experiments i ran it for 4 days. I ended with with 3.26 days worth of data (or Excel lines). Another time I ran the experiment for 3 days and ended with 1.7 days worth of data (or Excel lines).

I divided the actual time by the data recorded time (4 / 3.26) and got 1.23 & 1.76 respectively. Hence, I thought that my program cycle time was actually taking 1.23 seconds or 1.76 seconds for every cycle instead of the intended 1 second.

I ran the below code to measure the actual cycle time. I have read previously that opening a file, writing, and closing will take some time (up to 100 ms perhaps). However, my code measured less than 20 milliseconds for the entire process. Also my understanding of using millis() to delay instead of the delay() function ensures that a total of 1 second (referring to this particular example) has passed. For example, since opening, writing, and closing a file takes about 20 ms therefor and additional 980 ms must pass before entering the IF loop again. Where as, the delay() function will take a total of 1020ms.

NOTE: After running the below code you will see that on the serial monitor it will show less than 20ms for the opening, writing, and closing of the SD file. However, when checking the actual TEXT file on the SD card you will see about 980ms. I believe the reason is FileCloseTime is read after opening, writing, and closing the file, hence it is calculating the duration lapsed since this function was called last. 980ms is the difference between the 1000ms of the "IF" loop and the 20ms it takes to open, write, and close the SD file. Unless im wrong, this is further proof that it should be writing 86400 lines of data in a span of 24 hours.

Does anyone have any insight on where this delay is occurring and how to measure it?

Below is the code I used:

//FOR PROGRAM CYCLE TIME
const unsigned long eventInterval = 1000; //milli seconds
 unsigned long FileStartTime = 0; //milli seconds
 unsigned long FileCloseTime = 0; // milli seconds
unsigned long previousTime = 0;
unsigned long currentTime = 0;
//FUNCIONS
void writefile();
///for file program
#include <SD.h>
#include <SPI.h>
#include <Wire.h>

File myFile;
int pinCS = 10; // Pin 10 on Arduino Uno

void setup() 
{
  Serial.begin(115200);   // open serial over USB
  pinMode(pinCS, OUTPUT);
  
  if (SD.begin())
  {
    Serial.println("SD card is ready to use.");
  } else
  {
    Serial.println("SD card initialization failed");
    return;
  }

}

void loop() { 
  currentTime=millis();
  if (currentTime - previousTime >= eventInterval) {
    
writefile();

    Serial.print("Duration since last IF loop entrance <");Serial.print(currentTime-previousTime);Serial.println(">ms");
previousTime = currentTime;
 }
}

void writefile(){

  FileStartTime=millis();
  myFile = SD.open("test.txt", FILE_WRITE);
  if (myFile) {

    myFile.print("Duration since last IF loop entrance "); 
    myFile.print(currentTime-previousTime); //milli seconds time
    myFile.print("\t"); 
    myFile.print("Duration of file opening, writing, closing "); 
    myFile.println(FileStartTime-FileCloseTime); //milli seconds time
      
myFile.close(); // close the file   
      
  }
  // if the file didn't open, print an error:
  else {
    Serial.println("error opening test.txt");
  }
FileCloseTime=millis();
  Serial.print("write file function (total time) <");Serial.print(FileCloseTime-FileStartTime);Serial.println(">ms");
  
}
  

millis() has an error variation of a few uS. I think its 4 or 5 which when taken over the course of a few 1000's milliseconds may add up.

Let's walk through how your loop() works right now.

  1. update current time.
  2. check if 1 second has passed.... // time since last restart 1 sec.
  3. write file....... // time taken ~ 20 ms
  4. print a bunch of stuff...... // time taken ???? but Serial is Slowwww
  5. restart timer ....... // Total time since last restart 1.2 - 1.76 sec.

Maybe try restarting the timer before you do all that other stuff.

void loop() {
  currentTime = millis();
  if (currentTime - previousTime >= eventInterval) {
    
    previousTime = currentTime;
    writefile();

    Serial.print("Duration since last IF loop entrance <"); Serial.print(currentTime - previousTime); Serial.println(">ms");
  }
}

You're not right. When entering the loop, the OP stores the current time in a variable and then assigns a timer to this value. The duration of the loop itself with this approach does not affect the accuracy of the timer, unless the operations performed do not exceed 1 second in total.

Therefore your code modifications won't fix the problem.

previousTime = currentTime;

Try changing this to

previousTime += eventInterval;

millis() occasionally increments by 2 instead of 1, which will throw the exact interval off very slightly, but nowhere near what you are seeing.

Looking at the data on the SD card, what are you seeing as the intervals since the last IF loop?
Does the test code you posted exhibit the same loss of data samples as the data logger code?
Are you doing anything in the data logger code that disables interrupts, or has a delay in the library that gathers the data samples?
< edit > The DS18B20 temperature sensor is particularly bad for this, it has a read time of 750mS at the highest resolution.

I hear that a lot. :disguised_face:

it doesn't mean that you were right.
In this case, your logic is erroneous.

one could use a hardware timer to count once a second, a bit more accurate than the millis() cycle count.

An RTC set to output a 1Hz signal would be another option.

Thank you all for your input, it is much appreciated!

People reply fast in this forum and that is great!

I was scrutinizing my Arduino code, and didnt focus enough on my Excel functions :sweat_smile:

For now it seems to be an Excel function problem which has been fixed. The results make a lot more sense now :rofl:

If I find any future problems I will be sure to post again, but for now it seems to be ok :pray:

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.