Saving data .csv-file on SD-card works only for approx. 500 lines

[propy]

Hey, I'm trying to build a small plant watering system, with two moisture sensors and a pump (two senors just for more precise values). To get some information about the system, I implemented a standard Micro SD Card Adapter and logged the data every 2 seconds to a .csv-file with a small test code. I know that 2 seconds is a very short time span, if you want to let the thing run for a week, but that was only for testing purposes.

The problem I have now is, that everything works fine, until a certain point (I have no idea when that point is reached), where the data is no longer saved to the SD-card (but still logged in the serial monitor). In my example run 533 lines of data were saved. The second last entry in each row is the time from millis() and the last entry the time difference between the last two time steps. The only thing I could notice is the small increase in the time difference shortly after the failure (as visible in the Serial Monitor picture, in the fourth line).

.csv-file:

Serial Monitor:

Here is the full code:

#include <Arduino.h>
#include <SD.h>

#define PIN_SPI_CS 4
#define DELAY 2000

File myFile;

const char fileName[20] = "arduino.csv"; 

const int valAir = 800;
const int valWater = 330;

byte pumpPin = 5, buttonPin = 6;
byte sensorPin1 = A0, sensorPin2 = A1;
byte LEDPin = 8;
int sensorVal1, sensorVal1Percent;
int sensorVal2, sensorVal2Percent;
bool pumpVal, buttonVal, errorVal = 0;
char dataStr[100] = "";
char buffer[20];
long startTime, millisVal, prevMillisVal = 0;

void setup() {
  Serial.begin(9600);

  pinMode(pumpPin, OUTPUT);
  pinMode(LEDPin, OUTPUT);
  pinMode(sensorPin1, INPUT);
  pinMode(sensorPin2, INPUT);
  pinMode(buttonPin, INPUT_PULLUP);

  //CHECK SD-CARD
  if (!SD.begin(PIN_SPI_CS)) {
    Serial.println(F("SD CARD FAILED, OR NOT PRESENT!"));
    digitalWrite(LEDPin, 1);
    while (1);  // don't do anything more:
  }
  Serial.println(F("SD CARD INITIALIZED."));

  //CREATE FILE
  myFile = SD.open(fileName, FILE_WRITE);
  if (myFile) {
    myFile.println("PumpActive, AbsoluteSensorVal1, RelativeSensorVal1, AbsoluteSensorVal2, RelativeSensorVal2, AbsoluteSensorValMean, RelativeSensorValMean, millis, millisDiff");
  } else {
    Serial.print(F("SD Card: error on opening file"));
    errorVal = 1;
  }

}

void loop() {
  sensorVal1 = analogRead(sensorPin1);
  sensorVal1Percent = map(sensorVal1, valAir, valWater, 0, 100);
  sensorVal2 = analogRead(sensorPin2);
  sensorVal2Percent = map(sensorVal2, valAir, valWater, 0, 100);

  //ERROR CHECK
  digitalWrite(LEDPin, errorVal);
  if (errorVal == 1) {
    while(1);
  }

  //BUTTON (only for testing)
  buttonVal = !digitalRead(buttonPin);
  if (buttonVal == 1) {
    pumpVal = 1;
    startTime = millis();
  }
  if (millis() - startTime >= 6000 && pumpVal == 1) {
    pumpVal = 0;
  }
  
  //PUMP
  digitalWrite(pumpPin, pumpVal);

  //LOG
  logData(pumpVal, sensorVal1, sensorVal1Percent, sensorVal2, sensorVal2Percent);

  //DELAY
  delay(DELAY);
}

void logData(bool pumpVal, int sensorVal1, int sensorVal1Percent, int sensorVal2, int sensorVal2Percent) {
  dataStr[0] = 0;
  millisVal = millis();

  pumpVal == 1 ? strcat(dataStr, "1, ") : strcat(dataStr, "0, ");

  itoa(sensorVal1, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  itoa(sensorVal1Percent, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  itoa(sensorVal2, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  itoa(sensorVal2Percent, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  itoa((sensorVal1 + sensorVal2)/2, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  itoa((sensorVal1Percent + sensorVal2Percent)/2, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  ltoa(millisVal, buffer, 10);
  strcat(dataStr, buffer);
  strcat(dataStr, ", ");

  ltoa(millisVal - prevMillisVal, buffer, 10);
  strcat(dataStr, buffer);

  prevMillisVal = millisVal;

  //WRITE FILE
  if (myFile) {
    myFile.println(dataStr);
    myFile.flush();
  } else {
    Serial.print(F("SD Card: error on opening file"));
    errorVal = 1;
  }
  
  //SERIAL PRINT
  Serial.println(dataStr);
}

[b707]

Probably a

@propy

myFile.flush();

method works incorrectly.

Try to close and reopen the file, say, every 100 lines.

[jremington]

myFile.flush(); works fine. However, it does force the Arduino to write out the file buffer to SD card, then read it back again for the next update. That vastly increases the SD card error rate and current draw, while reducing the performance of the entire system. Use .flush() only once an hour or once a day.

You may be writing outside the bounds of the dataStr[] array with all those strcat() calls, but you won't know unless you keep track of the string length.

Use strncat() to prevent writing outside of bounds. Or, use snprintf to format the data and fill up dataStr in one function call.

[propy]

I did a test run with only flushing every 100 lines, and it seems to be working way better than before! Thank you!