Unable to write to SD card once program gets large enough

Using Arduino Storage
1

Hi,

I'm basically taking some analog input readings, doing some math, and then writing the results to an SD card. I also want to create a new file name for those results after every minute passes. To do this, I'm refencing an RTC and reading values from that to determine if it is a new minute or not.

Things work as expected up until a certain point. I'm not sure if it is the program storage space usage or dynamic memory usage (or something else entirely!), but at a certain point, the SD card functions stop working and neither create new files nor write to existing ones. I don't feel like I'm hitting the limits of the device -- I'm using an Uno and the failure happens at 74% program storage space and 81% dynamic memory usage -- but it feels like something is happening with the malloc() inherent to SD.open().

I've replicated this in multiple ways, but for the code I'm including here, I just added some debug serial print statements to cause the issue. If you comment out lines 177-186, the program works fine and writes correctly to the SD card, but if you leave those statements in, it fails to open or write to the SD card. I've caused this issue without using serial print statements, just by having more globals and doing more math as well, but this seemed like the best representation of the problem. I wouldn't think that these additions would have any effect on writing to the SD card, unless it has something to do with a string buffer issue or the malloc() portion of SD.open. Given the math that is going on in the background, the SD card only gets written to every ~10-15 seconds, so I'm not flooding the SD interface with requests or anything, and it fails on the very first write.

Has anyone run into an issue like this? Am I just missing something entirely (File.close() does not necessarily free the memory maybe? Something I'm not seeing in string manipulation where I'm causing a memory issue? etc.)?

For reference I'm just using an Uno with an SD card + RTC shield on top of it. Since I'm new, I cannot upload a file, but in the next post I'll paste the code in.

2 
#include <SPI.h>
#include <SD.h>
#include "RTClib.h"

#define SD_CS 10  //10 for big SD card reader

RTC_DS1307 rtc;
String fileName;
File dataFile;
DateTime time;

float const TEN_BIT = 1023.0;
float const VOLT_RANGE = 5.0;
const int AVGCOUNT = 15000;
const long tenMin = 1800000;
const long twoMin = 120000;

int bytesWritten = 0;
int loopCount = 0;
int v1_int_0, v2_int_0, v3_int_0;
float v1_0, v2_0, v3_0, a, b, c;
float d, e, h, i, j = 0.0, f;
float qAvg, qSum, tAvg, tSum = 0.0, rSum = 0.0, rAvg = 0.0, l = 0.0, uSum = 0.0;
unsigned long totalRunTime = 0;
unsigned long sampleTime = 0;
float g = 0.0, gAvg = 0.0, gSum = 0.0, m, n, o, p;
long tenMinCheck = 0;
long twoMinCheck = 0;

String currentYear = "0", currentMonth = "0", currentMinute = "0", previousMinute = "0";
bool newMinute = false;
String mode = "MODE A";
int k = 60;

void setup() {

  Serial.begin(9600);   
  //Initialize RTC
  if (! rtc.begin()) {
    Serial.println("Couldn't find RTC");
    Serial.flush();
    while (1) delay(10);
  }

  Serial.print("Initializing SD card...");
  if (!SD.begin(SD_CS)) {
    Serial.println("failed!");
  }
  else
  {
    Serial.println("OK!");
  }

  // start the SPI library:
  SPI.begin();
  //Set the max speed to 9MHz 
  SPI.beginTransaction(SPISettings(9000000, MSBFIRST, SPI_MODE0));

  //Grab the current time
  time = rtc.now();
  previousMinute = time.minute();

  currentYear = time.year() - 2000;
  currentMonth = time.month();
  if (currentMonth.toInt() < 10)
    currentMonth = "0" + currentMonth;
  currentMinute = time.minute();
  if (currentMinute.toInt() < 10)
    currentMinute = "0" + currentMinute;
    
  fileName = currentYear + currentMonth + currentMinute;
  fileName = fileName + ".CSV";

  // Create new file based on the year/month/minute
  //Open data file and write headers
  dataFile = SD.open(fileName, FILE_WRITE);
  dataFile.print("Power on time: ");
  dataFile.println(time.timestamp(DateTime::TIMESTAMP_FULL));
  dataFile.print("Current percent of the day, ms since last avg (");
  dataFile.print(AVGCOUNT);
  dataFile.println(" samples)");
  dataFile.close();
}

void loop() {

  if(loopCount == 0)
    totalRunTime = millis();

  v1_int_0 = analogRead(A0);
  v2_int_0 = analogRead(A1);
  v3_int_0 = analogRead(A2);

  // Convert the analog reading (which goes from 0 - 1023) to a voltage (0 - 5V):
  v1_0 = v1_int_0 * (VOLT_RANGE / TEN_BIT);
  v2_0 = v2_int_0 * (VOLT_RANGE / TEN_BIT);
  v3_0 = v3_int_0 * (VOLT_RANGE / TEN_BIT);

  a = v3_0 - v2_0;
  b = v1_0*4.8 - v2_0 - 2*(a);  
  c = v2_0/19.9;
  d = b/c;   
  e = b*c;
  f = (d - 100.6)/0.4;
  g = e/0.000104;
  h = e/(0.000104*(f - i));
  c = h*g*4*d*8.12*f*g;
  b = h*g*4*9888.245/e/f/g;
  l = c/h*(4.1782);
  m = (912.38012)*(h-f) + g;
  n = 567*a/308.83;


  

  if (loopCount == AVGCOUNT)
  {
        
    //Only update these averages that will be written to the SD card when in the correct mode
    if (mode == "MODE A")
    {
      gAvg = gSum/AVGCOUNT;
      gSum = g;      
      rAvg = rSum/AVGCOUNT;
      rSum = d;      
      tAvg = tSum/AVGCOUNT;
      tSum = f;      
      l = uSum/AVGCOUNT;
      uSum = h;      
    }
    
    if (mode == "MODE B")
    {
      i = j/AVGCOUNT;
      j = f;
    }
    qAvg = qSum/AVGCOUNT;
    qSum = e;

    sampleTime = totalRunTime;
    totalRunTime = millis();
    sampleTime = totalRunTime - sampleTime;
    
    //Check for 10 or 2 minute thresholds depending on the mode
    if (mode == "MODE A")
    {
      //Check if 10 mins have passed
      if (tenMinCheck >= tenMin)
      {
        tenMinCheck = 0;
        mode = "MODE B";
      }
      else if (tenMinCheck > 40000)
      {
        mode = "MODE A";
        tenMinCheck += sampleTime;
      }
      else
        tenMinCheck += sampleTime;
    }
    else
    {
      //Check if 2 mins have passed
      if (twoMinCheck >= twoMin)
      {
        twoMinCheck = 0;
        mode = "MODE C";
      }
      else
      {
        twoMinCheck += sampleTime;
        if (twoMinCheck >= (twoMin - 10000))
          mode = "MODE D";
      }
    }

    Serial.print("V1 = ");
    Serial.println(v1_0);
    Serial.print("V2 = ");
    Serial.println(v2_0);
    Serial.print("V3 = ");
    Serial.println(v3_0);
    Serial.print("b = ");
    Serial.println(b);
    Serial.print("d = ");
    Serial.println(d);

    //Grab the current time
    time = rtc.now();
    
    Serial.print("Hour = ");
    Serial.println(time.hour());
    Serial.print("Minute = ");
    Serial.println(time.minute());
    Serial.print("Seconds = ");
    Serial.println(time.second());
    

    //Check to see if we have crossed over into the next minute
    previousMinute = currentMinute;
    Serial.print("previousMinute = ");
    Serial.println(previousMinute);

    currentMinute = time.minute();    
    if (currentMinute.toInt() < 10)
      currentMinute = "0" + currentMinute;
    Serial.print("currentMinute = ");
    Serial.println(currentMinute);

    if (previousMinute != currentMinute)
    {
      //We crossed over into the next minute, so we need to create a new file
      Serial.println("Creating a new file due to the time");
        
      Serial.print("old fileName = ");
      Serial.println(fileName);
      currentYear = time.year() - 2000;
      currentMonth = time.month();
      if (currentMonth.toInt() < 10)
        currentMonth = "0" + currentMonth;
      currentMinute = time.minute();
      if (currentMinute.toInt() < 10)
        currentMinute = "0" + currentMinute;        

      fileName = currentYear + currentMonth + currentMinute;
      fileName = fileName + ".CSV";

      Serial.print("New filename = ");
      Serial.println(fileName);    
      newMinute = true;   

      //Open data file and write headers
      dataFile = SD.open(fileName, FILE_WRITE);
      if (dataFile)
        Serial.println("File opened");
      else
        Serial.println("File did not open");
      bytesWritten = dataFile.print("Power on time: ");
      Serial.print("Tried to write start of header, wrote ");
      Serial.print(bytesWritten);
      Serial.println(" bytes");
      dataFile.println(time.timestamp(DateTime::TIMESTAMP_FULL));

      dataFile.print("Current percent of the day, ms since last avg (");
      dataFile.print(AVGCOUNT);
      dataFile.println(" samples)");
      dataFile.close();
      delay(10);

    }

    //Write data to the file
    dataFile = SD.open(fileName, FILE_WRITE);
    /*
    if (newMinute)
    {
      dataFile.print("Power on time: ");
      dataFile.println(time.timestamp(DateTime::TIMESTAMP_FULL));
      dataFile.print("Current percent of the day, ms since last avg (");
      dataFile.print(AVGCOUNT);
      dataFile.println(" samples)");
      newMinute = false;
    }
    */
    dataFile.print(convertTime(time.hour(), time.minute(), time.second()));
    dataFile.print(",");
    dataFile.print(sampleTime);
    dataFile.print(",");
    dataFile.print(mode);
    dataFile.print(",");
    dataFile.print(qAvg, 6);
    dataFile.print(",");
    dataFile.print(i, 6);
    dataFile.print(",");
    dataFile.print(gAvg, 6);
    dataFile.print(",");

      
    dataFile.print(m);
    dataFile.print(",");
    dataFile.print(n);
    dataFile.print(",");
    dataFile.print(v3_0);
    dataFile.print(",");
    dataFile.print(rAvg);
    dataFile.print(",");
    dataFile.print(k);
    dataFile.print(",");
    dataFile.println(l);
    dataFile.close();

    loopCount = 0; 
  }
  else
  {
    loopCount++;
    qSum = qSum + e;
    if (mode == "MODE A")
      j = j + f;
    if (mode == "MODE C")
    {
      gSum = gSum + g;
      rSum = rSum + d;
      tSum = tSum + f;
      uSum = uSum + h;
    }
  }
  
  //Set the output based on the mode
  if (mode == "MODE A")
  {
    if (d < 150)
        k++; 
      else
        k--;
    if (k > 150)
      k = 150;
    if (k < 0)
      k = 0;
  }
  else
    k = 0;

  analogWrite(5, k);


}

//Convert the time from the RTC format to the Excel format (a percent of a day)
float convertTime(int hour, int minute, int second)
{
  float percentDay = 0.0;
  //Convert hours into days
  percentDay = hour/24.0;
  //Convert minutes into days
  percentDay += minute/1440.0;
  //Convert seconds into days
  percentDay += second/86400.0;

  return percentDay;    
}
3

I see now that the code doesn't have line numbers in it, so for reference, if you comment out this code:

    Serial.print("V1 = ");
    Serial.println(v1_0);
    Serial.print("V2 = ");
    Serial.println(v2_0);
    Serial.print("V3 = ");
    Serial.println(v3_0);
    Serial.print("b = ");
    Serial.println(b);
    Serial.print("d = ");
    Serial.println(d);

the program writes to the SD card properly, but leaving that in causes it to fail.

4

It sounds like you are running out of RAM with the SD card software allocating/trying to use more then you have. Try using the "F" macro on the print statements and see if that helps, if it does you just found the problem.

1 Like
5

Thanks for the quick response, and yes adding the (new to me) "F" macro just to those print statements I was commenting in and out seems to have fixed the problem! Awesome!

Looking into the F macro a bit and seeing how things work leads me to think about how to prevent this sort of things in the future, though.

So if SD.open() causes a malloc() of 512 bytes, and there's only 2k RAM, then I'm inferring that I would want to make sure that I'm only ever using 1536 bytes of what the IDE calls 'dynamic memory.' In the original (failing) code, I was using 1662 bytes of dynamic memory, and with the F macros in there, I'm now using 1634 bytes of dynamic memory.

My guess as to why it's not still failing is that the SD library is allocating the data it needs, but the program memory it's overwriting is not actually accessed again? I feel like to always be on the safe side, I should ensure to always have at least 512 bytes of dynamic memory free. Is that correct?

Thanks again!

6

I generally have a 8K x 8 FRAM on my systems. I can preload them with messages etc and that saves memory in both program FLASH and RAM. Catch you have to write another simple program to place the messages in it, then run your program. You communicate with the I2C versions same as external FLASH but no delays and it is good for over a billion cycles. Great for information that needs to be remembered during power failure. Also for logging as no write delay and it holds a lot of data which can be transferred to a SD card at will.

1 Like
7

This is why I build with Mega2560's. 2k of ram is just too limiting. 8k isn't much, but its better.

8

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.