SD card produces corrupt files

Hello all,

I have already read about similar problems in various threads, but unfortunately I can't pinpoint the problem with my setup.

As you can see i the pictures the sketch works fine at the beginning, all good. But after some time it will eventually turn into completely corrupted files, even the existing ones, which are then become unreadable.

My setup:

  • Arduino MEGA
  • Data-Logger shield with built in 3.3V regulator and level shifter Link
  • 8 Gb SD-card Fat32 Link
  • Arduino Base shield Link
  • MH19-C CO2 Sensor (hardware serial) (but not in use with the following sketch)
  • MQ-131, -136 - 137 (analog)
  • BME280 (I²C)
  • Multi-Gas Channel Sensor V1.0 (I²C)
  • lcd-display (I²C) (but not in use with the following sketch)
  • All powered with an external powersource via MB102 power supply module
    SD-corrupted files
    SD-files

My code is build to be very modular, and in fact is working, but for the sake of completeness and, well, even might be the problem after all.

I would be very grateful for your help, I am desperate to know why this keeps happening :frowning:

#include "RTClib.h" //SD-Karte
#include <SD.h>     //SD-Karte
#include <SPI.h>    //SD-Karte

#include <Wire.h>   //I2C, Multi-Gas-Channel, BME280, LCD-DIsplay
#include <Adafruit_Sensor.h>  // BME280
#include <Adafruit_BME280.h>  // BME280

#include "MutichannelGasSensor.h"

#include <MQUnifiedsensor.h> // MQ-131
#include <MQSinglesensor1_136.h> // MQ-136
#include <MQSinglesensor137.h> // MQ-137

#define SEALEVELPRESSURE_HPA (1017.40)// Hier TAGESaktuellen Luftdruck angeben für möglichst genaue Höhenbestimmung //BME280
Adafruit_BME280 bme; // I2C

#define ECHO_TO_SERIAL 1 // echo data to serial port

#define placa "Arduino UNO"
#define Voltage_Resolution 5
#define pin131 A0 //Analog input 0 of your arduino
#define pin136 A1 //Analog input 0 of your arduino
#define pin137 A2 //Analog input 0 of your arduino
#define Modell131 "MQ-131" //MQ131
#define Modell136 "MQ-136" //MQ136
#define Modell137 "MQ-137" //MQ137
#define ADC_Bit_Resolution 10 // For arduino UNO/MEGA/NANO
#define RatioMQ131CleanAir 1 //RS / R0 = 15 ppm
#define RatioMQ136CleanAir 1 //RS / R0 = 15 ppm
#define RatioMQ137CleanAir 1 //RS / R0 = 15 ppm
MQUnifiedsensor MQ131(placa, Voltage_Resolution, ADC_Bit_Resolution, pin131, Modell131);
MQSinglesensor1_ MQ136(placa, Voltage_Resolution, ADC_Bit_Resolution, pin136, Modell136);
MQSinglesensor MQ137(placa, Voltage_Resolution, ADC_Bit_Resolution, pin137, Modell137);

RTC_DS1307 RTC;     // define the Real Time Clock object
File logfile;
const int chipSelect = 10; // for the data logging shield, we use digital pin 10 for the SD cs line


unsigned long  LOG_StartMillis;
unsigned long currentMillis4;
const unsigned long LOG_interval = 3000;  //Interval for writing on the SD

void setup()
{
  Serial.begin(9600);  // for printing on serial monitor
  LOG_StartMillis = millis();         //initial start time
  setup_SDLogger();
  setup_BME280();
  setup_Multi_Gas_Channel();
  setup_MQ131();
  setup_MQ136();
  setup_MQ137();
  Serial.println();
}

//----------------------------------------------------------------------------------------------------------

void setup_SDLogger()
{
  Serial.print("Initializing SD card...");    // initialize the SD card
  pinMode(53, OUTPUT);      // make sure that the default chip select pin is set to output, even if you don't use it

  if (!SD.begin(chipSelect))
  {
    error("Card failed, or not present"); // see if the card is present and can be initialized:
  }
  Serial.println("card initialized.");

  char filename[] = "LOGGER00.CSV";      // create a new file
  for (uint8_t i = 0; i < 100; i++)
  {
    filename[6] = i / 10 + '0';
    filename[7] = i % 10 + '0';
    if (! SD.exists(filename))
    {
      logfile = SD.open(filename, FILE_WRITE);    // only open a new file if it doesn't exist
      break;  // leave the loop!
    }
  }

  if (! logfile)
  {
    error("couldnt create file");
  }

  Serial.print("Logging to: ");
  Serial.println(filename);

  Wire.begin();     // connect to RTC
  if (!RTC.begin())
  {
    logfile.println("RTC failed");
    Serial.println("RTC failed");
  }

  logfile.println("datetime,Temperatur in °C,Luftdruck in hPa,Luftfeuchtigkeit in %,O3 in ppb,MQ-131 R0 in KΩ,H2S in ppm,MQ-136 R0 in KΩ,NH3 in ppm,MQ-137 R0 in KΩ,CO in ppm,NO in ppm,NH3 in ppm,H2 in ppm");
  Serial.println("datetime,Temperatur in °C,Luftdruck in hPa,Luftfeuchtigkeit in %,O3 in ppb,MQ-131 R0 in KΩ,H2S in ppm,MQ-136 R0 in KΩ,NH3 in ppm,MQ-137 R0 in KΩ,CO in ppm,NO in ppm,NH3 in ppm,H2 in ppm");
}

void setup_BME280()
{
  bme.begin();
}

void setup_Multi_Gas_Channel ()
{
  gas.begin(0x04);
  gas.powerOn();
}

void setup_MQ131()
{
  MQ131.setRegressionMethod(1); //_PPB =  a*ratio^b
  MQ131.setA(9.4783); MQ131.setB(2.3348); // Configurate the ecuation values to get O3 concentration
  MQ131.init();
  MQ131.setRL(1);
  Serial.print("Calibrating please wait.");
  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ131.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ131.calibrate(RatioMQ131CleanAir);
  }
  MQ131.setR0(calcR0 / 10);
  Serial.println("  done!.");

  if (isinf(calcR0)) {
    Serial.println("Warning: Conection issue founded, R0 is infite (Open circuit detected) please check your wiring and supply");
    while (1);
  }
  if (calcR0 == 0) {
    Serial.println("Warning: Conection issue founded, R0 is zero (Analog pin with short circuit to ground) please check your wiring and supply");
    while (1);
  }
  /*****************************  MQ CAlibration ********************************************/
}
void setup_MQ136()
{
  MQ136.setRegressionMethod(1); //_PPM =  a*ratio^b
  MQ136.setA(0.1229); MQ136.setB(-3.873); // Configurate the ecuation values to get H2S concentration
  MQ136.init();
  MQ136.setRL136(47);
  Serial.print("Calibrating please wait.");
  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ136.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ136.calibrate(RatioMQ131CleanAir);
  }
  MQ136.setR0136(calcR0 / 10);
  Serial.println("  done!.");

  if (isinf(calcR0)) {
    Serial.println("Warning: Conection issue founded, R0 is infite (Open circuit detected) please check your wiring and supply");
    while (1);
  }
  if (calcR0 == 0) {
    Serial.println("Warning: Conection issue founded, R0 is zero (Analog pin with short circuit to ground) please check your wiring and supply");
    while (1);
  }
  /*****************************  MQ CAlibration ********************************************/
}

void setup_MQ137()
{
  MQ137.setRegressionMethod(1); //_PPM =  a*ratio^b
  MQ137.setA(0.1229); MQ137.setB(-3.873); // Configurate the ecuation values to get NH3 concentration
  MQ137.init();
  MQ137.setRL137(47);
  Serial.print("Calibrating please wait.");
  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ137.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ137.calibrate(RatioMQ131CleanAir);
  }
  MQ137.setR0137(calcR0 / 10);
  Serial.println("  done!.");

  if (isinf(calcR0)) {
    Serial.println("Warning: Conection issue founded, R0 is infite (Open circuit detected) please check your wiring and supply");
    while (1);
  }
  if (calcR0 == 0) {
    Serial.println("Warning: Conection issue founded, R0 is zero (Analog pin with short circuit to ground) please check your wiring and supply");
    while (1);
  }
  /*****************************  MQ CAlibration ********************************************/
}
void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  while (1);
}
//----------------------------------------- All setups above ------------------------------------------------------
void loop()
{
  update_MQ131();
  update_MQ136();
  update_MQ137();
  write_on_SD();
}



void update_MQ131()
{

  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ131.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ131.calibrate(RatioMQ131CleanAir);
  }
  MQ131.setR0(calcR0 / 10);
}

void update_MQ136()
{

  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ136.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ136.calibrate(RatioMQ136CleanAir);
  }
  MQ136.setR0136(calcR0 / 10);
}

void update_MQ137()
{

  float calcR0 = 0;
  for (int i = 1; i <= 10; i ++)
  {
    MQ137.update(); // Update data, the arduino will be read the voltage on the analog pin
    calcR0 += MQ137.calibrate(RatioMQ137CleanAir);
  }
  MQ137.setR0137(calcR0 / 10);
}

void write_on_SD()
{
  currentMillis4 = millis();
  DateTime now;
  now = RTC.now();    // fetch the time and log it
  if (currentMillis4 - LOG_StartMillis >= LOG_interval)
  {
    logfile.print(now.year(), DEC);
    logfile.print("/");
    logfile.print(now.month(), DEC);
    logfile.print("/");
    logfile.print(now.day(), DEC);
    logfile.print(" ");
    logfile.print(now.hour(), DEC);
    logfile.print(":");
    logfile.print(now.minute(), DEC);
    logfile.print(":");
    logfile.print(now.second(), DEC);
    logfile.print(", ");


    logfile.print(bme.readTemperature());
    logfile.print(", ");
    logfile.print(bme.readPressure() / 100.0F);
    logfile.print(", ");
    logfile.print(bme.readHumidity());
    logfile.print(", ");

    logfile.print(MQ131.readSensor());
    logfile.print(", ");
    logfile.print(MQ131.getR0());
    logfile.print(", ");

    logfile.print(MQ136.readSensor());
    logfile.print(", ");
    logfile.print(MQ136.getR0136());
    logfile.print(", ");

    logfile.print(MQ137.readSensor());
    logfile.print(", ");
    logfile.print(MQ137.getR0137());
    logfile.print(", ");

    logfile.print(gas.measure_CO());
    logfile.print(", ");
    logfile.print(gas.measure_NO2());
    logfile.print(", ");
    logfile.print(gas.measure_NH3());
    logfile.print(", ");
    logfile.print(gas.measure_H2());

    logfile.println();

    logfile.flush();

#if ECHO_TO_SERIAL
    Serial.print(now.year(), DEC);
    Serial.print('/');
    Serial.print(now.month(), DEC);
    Serial.print('/');
    Serial.print(now.day(), DEC);
    Serial.print(" ");
    Serial.print(now.hour(), DEC);
    Serial.print(":");
    Serial.print(now.minute(), DEC);
    Serial.print(":");
    Serial.print(now.second(), DEC);
    Serial.print(", ");

    Serial.print(bme.readTemperature());
    Serial.print(", ");
    Serial.print(bme.readPressure() / 100.0F);
    Serial.print(", ");
    Serial.print(bme.readHumidity());
    Serial.print(", ");

    Serial.print("O3 in ppb: ");
    Serial.print(MQ131.readSensor());
    Serial.print(", ");
    Serial.print("MQ-131 R0: ");
    Serial.print(MQ131.getR0());
    Serial.print(", ");

    Serial.print("H2S in ppm: ");
    Serial.print(MQ136.readSensor());
    Serial.print(", ");
    Serial.print("MQ-136 R0: ");
    Serial.print(MQ136.getR0136());
    Serial.print(", ");
    
    Serial.print("NH3 in ppm: ");
    Serial.print(MQ137.readSensor());
    Serial.print(", ");
    Serial.print("MQ-137 R0: ");
    Serial.print(MQ137.getR0137());
    Serial.print(", ");

    Serial.print("CO: ");
    Serial.print(gas.measure_CO());
    Serial.print(", ");
    Serial.print("NO2: ");
    Serial.print(gas.measure_NO2());
    Serial.print(", ");
    Serial.print("NH3: ");
    Serial.print(gas.measure_NH3());
    Serial.print(", ");
    Serial.print("H2: ");
    Serial.print(gas.measure_H2());
    Serial.print(", ");

    Serial.println();
    Serial.print(logfile.size());
    Serial.println();
    Serial.println("------------------------------");
#endif
    LOG_StartMillis = currentMillis4;  //IMPORTANT to save the start time of the current LED state.
  }
}

float Sensorreading(char type)
{
  float value;
  float temp = bme.readTemperature();
  if (type == 'H')
  {
    value = bme.readHumidity();// read humidity
  }
  else if (type == 'P')
  {
    value = bme.readPressure() / 100.0F; // read pressure
  }

  else if (type == 'Q')
  {
    value = gas.measure_CO(); // read CO Multi-Gas-Cannel
  }
  else if (type == 'W')
  {
    value = gas.measure_NO2(); // read NO2 Multi-Gas-Cannel
  }
  else if (type == 'N')
  {
    value = gas.measure_NH3(); // read NH3 Multi-Gas-Cannel
  }
  else if (type == 'M')
  {
    value = gas.measure_H2(); // read H2 Multi-Gas-Cannel
  }
  else
  {
    value = bme.readTemperature();// read temperature
  }
  return value;
  // Bereits verwendete Buchstaben: H, G, N, M, P, Q, W
}

But, surely, you know when it began to appear? Was it working properly from the beginning and suddenly went bad?
Paul

Good question,

not really, I never thought about a problem like this, so as I went on i changed my setup and program, but never let it run for several hours or a few days.

What I know right now, it works for at least ~15 min but after 2 hours it already messes things up.

First I would suggest using a HEX file dump program to print the data files and see if you can find where the first error occurs and exactly where it goes wrong. Any particular field or where in a record?

Paul

That's exactly what I thought, but unfortunately the program I used (HEX-Editor MX or Notepad++) count open the corrupted files :frowning: but I am definitely no expert on this kind of things.

In the meantime I tried to keep it plugged to my Pc with open serial monitor, in which everything seemed absolutely fine. But the files on the sd told another story...

SD-corrupted files 2

Does you hex dump program work with files that are created during the 15 minutes when the program is working correctly?
Paul

yes they do, but as long they are not corrupted you can easily open them. These files are .csv / .txt i would upload them if I could

How exactly do you have the data logger shield wired to the arduino? The shield appears to be designed for an UNO, with the SPI bus on pins 11/12/13, and CS on pin 10, while a mega has SPI on pins 50/51/52 and CS on pin 53. Your code sets pin 53 as output, then uses pin 10 for the CS of the SD card.

Hello david,

sorry for not including this information right from the beginning.

I used this tutorial on how to get a rev B sd card shield running with an arduino mega. Link

Pin 50 -> MISO Pin 12
Pin 51 -> MOSI Pin 11
Pin 52 -> SCK Pin 13
Pin 53 -> CS Pin 10

Pin A4 (bent out) -> SDA
Pin A5 (bent out) -> SCL

If you have pin 53 connected for the CS, then the variable chipSelect in the sketch should be changed from 10 to 53.