.csv File being created on SD card, no data within file

Hello, I have been trying to fix this code for over a week and would now like to turn this forum for some help. Just a general overview, I am using an Arduino Mega for a project to store water quality data from Total dissolved solid (TDS), DO, turbidity and temperature sensors. 3 of the 4 are analog, one digital. I have all the sensors working and reading data into the serial monitor, but nothing shows up in the .csv file "LOGGER00X" that gets created with each run. Right now I have multiple headers set up but am only trying to log data from one sensor to start (the TDS value). Code verifies fine for my board. I tried the SD datalogger example in the IDE and that worked fine (created a file and saved the data to it), so I know it shouldn't be an issue with my hardware. I am very new to this so please don't judge my code if there is a glaring / super obvious problem causing this for me. TYIA

Hardware:
-Arduino Mega 2560 R3 board stacked with
-DF robot expansion shield for the mega board (https://www.robotshop.com/en/dfrobot-mega-io-expansion-shield-arduino-mega.html?gclid=CjwKCAjw4qCKBhAVEiwAkTYsPNrRPs75XVy7t8GMiWq4WbdRSCeMp8QzLWiwh3MvcCZgPFWvWQWDIhoCItkQAvD_BwE)
-Adafruit datalogging shield stacked on top of the expansion shield (https://cdn-learn.adafruit.com/downloads/pdf/adafruit-data-logger-shield.pdf)

Code:

//Mega board with TDS on A11, DO on A12, Turbidity on A13, Temp probe on Digital pin 14
//Notes: As of 9/20/2021 cannot get data to print to SD card. Files are created but empty


/*** SD Code***/
#include "SD.h"
//#include <SPI.h>
#include <Wire.h>
#include "RTClib.h"
#define LOG_INTERVAL  100 // mills between entries
#define SYNC_INTERVAL 100 // mills between calls to flush() - to write data to the card
uint32_t syncTime = 0; // time of last sync()
#define ECHO_TO_SERIAL   1 // echo data to serial port
RTC_PCF8523 RTC_SD; // define the Real Time Clock object- need to refer to the one on the mircrocontroller ATmega2560

// for the DFrobot Mega shield w/ micro SD , we use digital pin 53 on the Mega (10 for normal boards and SD shield) for the SD cs line
const int chipSelect = 10;

/*** TDS Code***/
#define TdsSensorPin A11
const int VREF = 5.0;      // analog reference voltage(Volt) of the ADC (Analog to Digital Converter)
const int SCOUNT = 30; // sum of sample point
int analogBuffer[SCOUNT];    // store the analog value in the array, read from ADC
int analogBufferTemp[SCOUNT];
int analogBufferIndex = 0, copyIndex = 0;
float averageVoltage = 0, tdsValue = 0, temperature = 25;

/*** DO Sensor Code***/
#include <Arduino.h>

#define DO_PIN A12

#define VREF 5000    //VREF (mv)
#define ADC_RES 1024 //ADC Resolution

//Single-point calibration Mode=0
//Two-point calibration Mode=1
#define TWO_POINT_CALIBRATION 0

#define READ_TEMP (25) //Current water temperature ℃, Or temperature sensor function--- should this be using info from temp sensor?

//Single point calibration needs to be filled CAL1_V and CAL1_T
#define CAL1_V (1600) //mv
#define CAL1_T (25)   //℃
//Two-point calibration needs to be filled CAL2_V and CAL2_T
//CAL1 High temperature point, CAL2 Low temperature point
#define CAL2_V (1300) //mv
#define CAL2_T (15)   //℃

const uint16_t DO_Table[41] = {
  14460, 14220, 13820, 13440, 13090, 12740, 12420, 12110, 11810, 11530,
  11260, 11010, 10770, 10530, 10300, 10080, 9860, 9660, 9460, 9270,
  9080, 8900, 8730, 8570, 8410, 8250, 8110, 7960, 7820, 7690,
  7560, 7430, 7300, 7180, 7070, 6950, 6840, 6730, 6630, 6530, 6410
};

uint8_t Temperaturet;
uint16_t ADC_Raw;
uint16_t ADC_Voltage;
uint16_t DO;

int16_t readDO(uint32_t voltage_mv, uint8_t temperature_c)
{
#if TWO_POINT_CALIBRATION == 0
  uint16_t V_saturation = (uint32_t)CAL1_V + (uint32_t)35 * temperature_c - (uint32_t)CAL1_T * 35;
  return (voltage_mv * DO_Table[temperature_c] / V_saturation);
#else
  uint16_t V_saturation = (int16_t)((int8_t)temperature_c - CAL2_T) * ((uint16_t)CAL1_V - CAL2_V) / ((uint8_t)CAL1_T - CAL2_T) + CAL2_V;
  return (voltage_mv * DO_Table[temperature_c] / V_saturation);
#endif
}
/*** Temp Probe Code***/
#include <OneWire.h>

int DS18S20_Pin = 14; //DS18S20 Signal pin on digital 14


//Temperature chip i/o
OneWire ds(DS18S20_Pin);

/*** SD Code***/
// the logging file
File logfile;
void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  while (1);
}

///////////////////////////////////////////////////////////////////////////////////////////////////////////
void setup()
{
  Serial.begin(115200); //Same BAUD rate for all sensors
  //SD Code//
  Serial.println("Serial Up!");
  // initialize the SD card
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);

  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    Serial.println("Card failed, or not present");
    // don't do anything more:
    return;
  }
  Serial.println("card initialized.");

  // create a new file
  char filename[] = "LOGGER00.CSV";
  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = i / 10 + '0';
    filename[7] = i % 10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE);
      break;  // leave the loop!
      logfile.close(); //Do I need to remove this to stop it from closing every time?
    }
  }
  if (! logfile)
  {
    error("couldnt create file");
  }
  Serial.print("Logging to: ");
  Serial.println(filename);


  //my code, not JJ's

  RTC_SD.begin(); //given error message about rtc being pointer type
  RTC_SD.adjust(DateTime(__DATE__, __TIME__));  //Set up the date/time to match computer
  //RTC_SD.setTime();//write time to chip
  //
  Wire.begin();
  if (!RTC_SD.begin()) {
    logfile.println("RTC failed");
#if ECHO_TO_SERIAL
    Serial.println("RTC failed");
#endif  //ECHO_TO_SERIAL
  }
  logfile.println("millis,datetime,TDS,DO temp, DO ADC, DO Volt,DO, Turb, Water temp");
#if ECHO_TO_SERIAL
  Serial.println("millis,datetime,TDS,DO temp, DO ADC, DO Volt,DO, Turb, Water temp");

#endif ECHO_TO_SERIAL// attempt to write out the header to the file
  //  if (logfile.write_error || !logfile.sync()) {
  //    error("write header");

  //  // If you want to set the aref to something other than 5v
  //  analogReference(EXTERNAL);
  //  pinMode(13, OUTPUT);
  //  ec.begin();
  //  pinMode(TdsSensorPin, INPUT);
  //
  //TDS Code
  pinMode(TdsSensorPin, INPUT); // No DO or turbidity code needed in Set up
}
/////////////////////////////////////////////////////////////////////////////////////////////////////////
void loop()
{
  //SD code//
  DateTime now; now = RTC_SD.now(); // this differs from JJ's code

  // delay for the amount of time we want between readings
  delay((LOG_INTERVAL - 1) - (millis() % LOG_INTERVAL));

  // log milliseconds since starting
  uint32_t m = millis();
  logfile.print("ms since start: ");
  logfile.print(m);           // milliseconds since start
  logfile.println(", ");
#if ECHO_TO_SERIAL
  Serial.print("ms since start: ");
  Serial.println(m);         // milliseconds since start
#endif
  //    if (! SD.exists(filename)) {
  //       only open a new file if it doesn't exist
  logfile = SD.open("TESTFILE.CSV", FILE_WRITE);
  //      break;  // leave the loop! Unable to open and write to file
  //    }
  logfile.write("STARTING THE LOG FILE");
  // fetch the time
  //  now = DateTime RTC.now();
  // log time
  //  logfile.print(now); // seconds since 2000
  logfile.print("datetime:");
  logfile.print(now.year(), DEC);
  logfile.print("/");
  logfile.print(now.month(), DEC);
  logfile.print("/");
  logfile.print(now.day(), DEC);
  logfile.print(" ");
  logfile.print(now.hour(), DEC);
  logfile.print(":");
  logfile.print(now.minute(), DEC);
  logfile.print(":");
  logfile.print(now.second(), DEC);
  logfile.println(", ");
#if ECHO_TO_SERIAL
  //  Serial.print(now()); // seconds since 2000
  Serial.print("datetime: ");
  Serial.print(now.year(), DEC);
  Serial.print("/");
  Serial.print(now.month(), DEC);
  Serial.print("/");
  Serial.print(now.day(), DEC);
  Serial.print(" ");
  Serial.print(now.hour(), DEC);
  Serial.print(":");
  Serial.print(now.minute(), DEC);
  Serial.print(":");
  Serial.println(now.second(), DEC);
#endif //ECHO_TO_SERIAL 
  // TDS Code
  static unsigned long analogSampleTimepoint = millis();
  if (millis() - analogSampleTimepoint > 40U)  //every 40 milliseconds,read the analog value from the ADC
  {
    analogSampleTimepoint = millis();
    analogBuffer[analogBufferIndex] = analogRead(TdsSensorPin);    //read the analog value and store into the buffer
    analogBufferIndex++;
    if (analogBufferIndex == SCOUNT)
      analogBufferIndex = 0;
  }
  static unsigned long printTimepoint = millis();
  if (millis() - printTimepoint > 800U)
  {
    printTimepoint = millis();
    for (copyIndex = 0; copyIndex < SCOUNT; copyIndex++)
      analogBufferTemp[copyIndex] = analogBuffer[copyIndex];
    averageVoltage = getMedianNum(analogBufferTemp, SCOUNT) * (float)VREF / 1024.0; // read the analog value more stable by the median filtering algorithm, and convert to voltage value
    float compensationCoefficient = 1.0 + 0.02 * (temperature - 25.0); //temperature compensation formula: fFinalResult(25^C) = fFinalResult(current)/(1.0+0.02*(fTP-25.0));
    float compensationVolatge = averageVoltage / compensationCoefficient; //temperature compensation
    tdsValue = (133.42 * compensationVolatge * compensationVolatge * compensationVolatge - 255.86 * compensationVolatge * compensationVolatge + 857.39 * compensationVolatge) * 0.5; //convert voltage value to tds value
    //Serial.print("voltage:");
    //Serial.print(averageVoltage,2);
    //Serial.print("V   ");
    Serial.print("TDS Value:");
    Serial.print(tdsValue, 0);
    Serial.println("ppm");
  }
  // DO Code
  Temperaturet = (uint8_t)READ_TEMP;
  ADC_Raw = analogRead(DO_PIN);
  ADC_Voltage = uint32_t(VREF) * ADC_Raw / ADC_RES;

  Serial.print("DO Calibration Temperature (C):\t" + String(Temperaturet) + "\t");
  Serial.print("ADC RAW:\t" + String(ADC_Raw) + "\t");
  Serial.print("ADC Voltage:\t" + String(ADC_Voltage) + "\t");
  Serial.print("DO:\t" + String(readDO(ADC_Voltage, Temperaturet)) + "\t");
  Serial.println(" ug/L");

  // Turbidity Code
  int sensorValue = analogRead(A13);//
  float voltage = sensorValue * (5.0 / 1024.0); // Convert the analog reading (which goes from 0 - 1023) to a voltage (0 - 5V):
  //Serial.println(voltage); // print out the value you read:
  Serial.print("Turbidity:" + String(voltage) );
  Serial.println(" NTU");//add units

  //Temp Probe Code
  float temperature = getTemp();
  Serial.print("Water Temperature (C):");
  Serial.println(temperature);
  Serial.print("\n");

  delay(1000);
}
int getMedianNum(int bArray[], int iFilterLen)
{
  int bTab[iFilterLen];
  for (byte i = 0; i < iFilterLen; i++)
    bTab[i] = bArray[i];
  int i, j, bTemp;
  for (j = 0; j < iFilterLen - 1; j++)
  {
    for (i = 0; i < iFilterLen - j - 1; i++)
    {
      if (bTab[i] > bTab[i + 1])
      {
        bTemp = bTab[i];
        bTab[i] = bTab[i + 1];
        bTab[i + 1] = bTemp;
      }
    }
  }
  if ((iFilterLen & 1) > 0)
    bTemp = bTab[(iFilterLen - 1) / 2];
  else
    bTemp = (bTab[iFilterLen / 2] + bTab[iFilterLen / 2 - 1]) / 2;
  return bTemp;
}
float getTemp() {
  //returns the temperature from one DS18S20 in DEG Celsius

  byte data[12];
  byte addr[8];

  if ( !ds.search(addr)) {
    //no more sensors on chain, reset search
    ds.reset_search();
    return -1000;
  }

  if ( OneWire::crc8( addr, 7) != addr[7]) {
    Serial.println("CRC is not valid!");
    return -1000;
  }

  if ( addr[0] != 0x10 && addr[0] != 0x28) {
    Serial.print("Device is not recognized");
    return -1000;
  }

  ds.reset();
  ds.select(addr);
  ds.write(0x44, 1); // start conversion, with parasite power on at the end

  byte present = ds.reset();
  ds.select(addr);
  ds.write(0xBE); // Read Scratchpad


  for (int i = 0; i < 9; i++) { // we need 9 bytes
    data[i] = ds.read();
  }

  ds.reset_search();

  byte MSB = data[1];
  byte LSB = data[0];

  float tempRead = ((MSB << 8) | LSB); //using two's compliment
  float TemperatureSum = tempRead / 16;
  // float temperature = getTemp();

  return TemperatureSum;

  delay(1000); //wait 1 seconds writing to SD file
  logfile.print(", ");
  logfile.print(tdsValue);
  logfile.print(", ");
#if ECHO_TO_SERIAL
  Serial.print(", ");
  Serial.println(tdsValue);
#endif //ECHO_TO_SERIAL
  ////////////////
  Serial.print("Closing logfile");
  logfile.close();
}

Have you tested the SD subsystem with the simple examples from the SD library?

I'm not sure what you mean by this. I ran the example in from the SD library for datalogging and have also formatted my card accordingly. Is there a specific example you're referring to?

Do you see your "closing log file" message?
Paul

@Paul_KD7HB No it doesn't appear in the Serial monitor. I didn't really notice before as it was autoscrolling.

Use a Serial Terminal program (i.e. NOT the Arduino IDE) that logs everything to a file, that way you can review it afterwards and dont miss important stuff.

I use TeraTerm.

Now you know where to begin debugging.
Paul

@srnet thank you. I downloaded Tera Term and "logged" a file but it gave me the same information as I was seeing on my Serial monitor. Is there a different file or metadata I should be looking for?

@Paul_KD7HB Thank you Paul. I don't have any set amount of time for my program to run, the data is just collected continuously... do I need to have an ending point to then close the file? Or can the file continuously open and close as data logs?

The point is, the TeraTerm file data are permanently recorded so you don't miss anything.

do I need to have an ending point to then close the file? Or can the file continuously open and close as data logs?

To get the last data block, you have to close the SD file. You do not want to be continuously opening and closing the file as that slows the process way down and also leads to read/write errors.

Open the file once in setup(), and close it only when you are done writing. Take this out of loop():

  //    if (! SD.exists(filename)) {
  //       only open a new file if it doesn't exist
  logfile = SD.open("TESTFILE.CSV", FILE_WRITE);
  //      break;  // leave the loop! Unable to open and write to file
  //    }

Well now you know you did not miss anything, which is the point of using something like Teraterm.

Ah, I see. Thank you for the suggestion!

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.