Arduino script and SD card library?

Hello!
I am working on a pretty big script using the SD card library and datalogging info from sensors. At present, I have a working code, but when I try to add more code, it stops working. That is, even when I add an empty: if(TRUE) {} to the end of my void loop(), then the program stops working when I upload it. The only thing I could figure is that I have used up all my memory for the program?? But at the bottom of my arduino window, it says "Binary sketch size: 22582 bytes (of a 32256 byte maximum)", so it looks like I am not using it all...

Any ideas? Any hidden reasons for this? Does the SD library take it's SD buffer out of this memory, for example? (I would think it RAM, but...)

Thanks!
-Nick

There are three kinds of memory. The compiler is reporting on the use of one of them. You may well be running out of SRAM memory even though the sketch does not fill the Flash memory space.

Does the SD library take it's SD buffer out of this memory, for example?

Yes, the SD library uses a lot of SRAM (512 bytes+), of which you only have 2K.

There are ways to move stuff from SRAM to Flash, but we need to see your code to see if that will help.

Thanks for the reply. Here is most of the code, with some omitted so it fits in the message size limit:

#include <SD.h>
#include <SPI.h>
#include <Wire.h>
#include "RTClib.h"
#include <DHT22.h>
#include <stdlib.h>


RTC_DS1307 RTC;

File logfile; //declare the name of the logging file
//char current_file[]="UPODXX/MMDDYYYY.txt";
char current_file[]="XXMMDDYY.txt";  //make sure to check first_date value 
char model[]="UPOD00"; //UPOD model indicator


void setup()

  {
   
   //attachInterrupt(1,calibrate,RISING);  //signal interrupt for calibration button
   
   pinMode(CS_Uno, OUTPUT);
   pinMode(CS_ADC_1, OUTPUT);
   digitalWrite(CS_ADC_1,HIGH); //deselect ADC initially
   pinMode(CS_ADC_2, OUTPUT);
   digitalWrite(CS_ADC_2,HIGH); //deselect ADC initially
   
   pinMode(red_led,OUTPUT);
   pinMode(green_led,OUTPUT);
   pinMode(flag_led,OUTPUT);
   
   Serial.begin(9600);

   Wire.begin();
   RTC.begin();
   
   SPI.begin();
   SPI.setDataMode(0);
   SPI.setBitOrder(MSBFIRST);
   SPI.setClockDivider(SPI_CLOCK_DIV16);
   //delay(200); //allow initialization
   
   if (RESET_LOCAL_TIME)
    {
     RTC.adjust(DateTime(__DATE__, __TIME__)); 
    }
   
   
   if (! SD.begin(CS_Uno)) 
    {
      if(ECHO_DEBUG)
      {
        Serial.println("*Card initialization failed*");
      }
      
      //while(1) //lock into loop with red/green sync flashing at 1Hz for 10 sec, then reboot system 
      for(int i=1;i<=10;i++)
      {
       digitalWrite(green_led, HIGH);
       digitalWrite(red_led,HIGH);
       delay(500);
       digitalWrite(green_led,LOW);
       digitalWrite(red_led,LOW);
       delay(500);
      }
      soft_reset();
    }
    
   if (! RTC.isrunning()) 
   {
       if(ECHO_DEBUG)
      {
        Serial.println("*RTC is not working*");
      }
   
       while(1) //Flash red/green in alternate at 1Hz if RTC not working
       {
        digitalWrite(green_led,HIGH);
        digitalWrite(red_led,LOW);
        delay(500);
        digitalWrite(green_led,LOW);
        digitalWrite(red_led,HIGH);
        delay(500); 
       } 
    }    
    
    DateTime now=RTC.now();
    
    if(ECHO_DEBUG)
    {
     Serial.println();
     Serial.print("Current File: ");
     Serial.println(current_file); 
    }
    
     if(! SD.exists(current_file))
    {
      logfile=SD.open(current_file, FILE_WRITE);
      char Header[]= "Analog2  Analog3  Analog4  Analog5";
      logfile.println(Header);
      logfile.flush();
    }
    //if(SD.exists(current_file))
    else
    {
      logfile=SD.open(current_file, FILE_WRITE);
    }
    
    if(!logfile)  
    {
     if(ECHO_DEBUG)
      {
        Serial.println("*File cannot be found or opened on card*");
      }
      
      //while(1) //lock into loop with red/green sync flashing at 1Hz
      for(int i=1;i<=10;i++)
      {
       digitalWrite(green_led, HIGH);
       digitalWrite(red_led,HIGH);
       delay(500);
       digitalWrite(green_led,LOW);
       digitalWrite(red_led,LOW);
       delay(500);
      }
      soft_reset();
    }
  }
  
  void loop()
  {
    
    DateTime now=RTC.now();
    
    if(ECHO_DEBUG)
    {
     Serial.println("Sampling Sensors..."); 
    }
    
    digitalWrite(green_led,HIGH);  //flash green light when sampling
    delay(50);
    digitalWrite(green_led,LOW);
    
    //Sample sensors and write to buffer
    ADCcall(ADC1, CS_ADC_1);
    ADCcall(ADC2,CS_ADC_2);
    
    myRHT03.readData(); //read the RHT03 T and Rh
    
    //add values to space delimited data line
    logfile.print(model); //model name
    logfile.print("  ");
    logfile.print((int)now.year()); //current year
    logfile.print("/");
    logfile.print((int)now.month()); //current month
    logfile.print("/");
    logfile.print((int)now.day()); //current day
    logfile.print("  ");
    logfile.print((int)now.hour()); //curent hr
    logfile.print(":");
    logfile.print((int)now.minute()); //current minute
    logfile.print(":");
    logfile.print((int)now.second()); //current second
    logfile.print("  ");
    logfile.print(now.unixtime()); //current unixtime; time since midnight 1/1/1970 (seconds)
    logfile.print("  ");
    logfile.print(ADC2[1]); //BAT voltage
    logfile.print("  ");
    logfile.print(analogRead(A1)); //Baseline VOC sensor
    logfile.print("  ");
    logfile.print(analogRead(A0)); //CO2 sensor
    logfile.print("  ");
    logfile.print(analogRead(A2)); //CO electrolytic sensor
    logfile.print("  ");
    
    dtostrf(myRHT03.getTemperatureC(),6,2,RHT03_buffer); //Temp internal
    logfile.print(RHT03_buffer);
    logfile.print("  ");
    dtostrf(myRHT03.getHumidity(),6,2,RHT03_buffer); //Rh internal
    logfile.print(RHT03_buffer);

    
    logfile.println();
    flush_crit++; //index counter up for flus criteria
   
    if(flush_crit>=cycles_b4_flush)
    { 
      digitalWrite(red_led,HIGH);
      logfile.flush();
      
      flush_crit=0;
      delay(200);
      digitalWrite(red_led,LOW);
      
      if(ECHO_DEBUG)
      {
       Serial.println("Flushing SD card buffer"); 
       Serial.println("Closing and reopening file");
       Serial.println();
      }
      
        logfile.close(); //close and reopen logfile to check SD card presence
        current_file[(first_date+2)]=now.day()/10+'0';
        current_file[(first_date+3)]=now.day()%10+'0'; //edit current file
        
        
        if(! SD.exists(current_file))
        { 
         /* logfile=SD.open(current_file, FILE_WRITE);
          char Header[]="Model  YYYY/MM/DD  Hour  Minute  Second  UnixTime  "
          "Op_voltage  Baseline  CO2  CO Fig1  Fig2  e2v_O3  e2v_NO2  e2v_1  "
          "e2v_2  e2v_3  e2v_4  Temp_outside  Temp_inside  Rh  Light  Analog1  Analog2  Analog3  Analog4  Analog5";
          logfile.println(Header);
          logfile.flush();  */
          soft_reset();
        }
        
        //if(SD.exists(current_file))
        else
        {
          logfile=SD.open(current_file, FILE_WRITE);
        }
        
        if(!logfile)
        {
         soft_reset();
        }
             
    } //end 'flush' loop
    
    //if(calibrate_flag) //enter calibration procedure if calibration button is hit; note: 'calibrate_flag' set with signal interrupt
   // {
      /*for(int i=1; i<=int(CALIBRATION_DURATION*60000./500.); i++)
     {
      digitalWrite(red_led,LOW);
      digitalWrite(green_led,LOW);
      digitalWrite(flag_led,HIGH);
      delay(150);
      digitalWrite(HIGH,LOW);
      digitalWrite(green_led,LOW);
      digitalWrite(flag_led,LOW);
      delay(150);
      digitalWrite(red_led,LOW);
      digitalWrite(green_led,HIGH);
      digitalWrite(flag_led,LOW);
      delay(150);
     }*/
     
   // }
    
    delay(1000/SAMPLE_RATE); 
  }
 
  
  
  
  void ADCcall(int Figaro[], int pin)
  {
    const byte ADC_ch[8]={B1000,B1001,B1010,B1011,B1100,B1101,B1110,B1111};  //channel designations
    const byte start_byte= B00000001; //initial byte to send to start ADC transmission
  
    digitalWrite(pin,LOW);
    delay(10);
    digitalWrite(pin,HIGH); //cycle ADC to reset if it was in LOW initially
    delay(10);
    
    
    for (int i=0; i<8; i++)
      {
      digitalWrite(pin,LOW); //activate ADC chip
    
      byte channel_byte=ADC_ch[i]<<4; //determine channel to read from
    
      SPI.transfer(start_byte);
      byte byte1=SPI.transfer(channel_byte);
      byte byte2=SPI.transfer(B00000000);
    
      Figaro[i]= int(((byte1 & B00000011)<<8 | byte2) ); //mask first 6 bits of 'byte1' and concatenate w/ 'byte2'
      
      digitalWrite(pin,HIGH); //deselect ADC
      } 
  }
  
 void soft_reset() //software reset; bring script to beggining of reset loop; internal watchdog on atmega
  { //http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1273155873/all
    asm volatile ("  jmp 0");
  }
  

/*void calibrate()
{
  calibrate_flag=true;
 Serial.println("CALIBRATE");
} 
*/

I'll have to look in to the different types of memory, and how different parts of the sketch are partitioned/allocated/using different types of memory. Intuitively, wouldn't the if() and other logical loops be compiled and stored in the machine code and not RAM? It doesn't make sense to me that adding an extra if() loop to my code would use up all the RAM.

The if(calibrate_flag) {} loop is the one that causes the system to stop working. Even with the inside of the {} commented out, it still messes up the operations.

The code will also stop working if I attach a signal interrupt to one of the pins; best I know this is part of the register?

Thanks,
-Nick

You have quite a few strings in there that are using RAM. Look at progmem and the F() macro to see how to move them to flash memory. Also, grab one of the freemem functions posted often in the forums or the playground. It'll tell you how much of your RAM you're using.

Great, thanks for the info! I'll look in to shifting some of the data to flash memory rather than RAM. If it is not too much trouble, would someone mind pointing out some examples of

a few strings in there that are using RAM

. Presumably when I use the function logfile.print(), the data is stored in RAM until it is flushed from the buffer, right? (I assume the 'buffer' must be made of RAM..,?). But what other 'strings' exactly would be using RAM? To the best of my knowledge, the character strings (if that is what is meant by 'strings'), will be stored in a register, unless they are qualified as 'volatile', right? I'll definitely read in to this to expand my understanding of it all, but some quick insight so I can get back on the road would be really appreciated.
Thanks,
-Nick

Ah... it appears that any variable will be in SRAM as it is manipulated throughout the program execution? Correct? So perhaps I could also save some SRAM by using #define and const on some of my constants?

Ah... it appears that any variable will be in SRAM as it is manipulated throughout the program execution? Correct?

Yes, but, as was mentioned, you have a lot of constant strings, in the Serial.print() functions. Those can be used straight out of Flash, without ever being transferred to SRAM, with the F() macro.

For instance, change:
Serial.println("Card initialization failed");
to
Serial.println(F("Card initialization failed"));
and watch your SRAM needs drop.

Okay, great! I've been reading a little bit about variable memory allocation, etc... and it isn't clear to me exactly how variables stored in flash are manipulated. If I use F(), and the string/integer/whatever is constant, then the Serial.print(F("...")) will just write the string from flash memory. If, however, it is a variable, and I use the progmem functions, then doesn't the variable (stored in Flash) have to be 'loaded' in to RAM to be manipulated, then stored back in flash after manipulation? Is this how the controller/flash program/machine code carries out the process? (or does a progmem stored variable never "leave" the flash memory?).

I only ask because, if that is the case, then the amount of RAM being used will vary depending on the process and which variables are being read/written from flash to RAM. If so, then wouldn't your RAM needs change throughout a program execution? I also haven't looked in to the 'freemem' functions yet, but will they tell you how much RAM is being used THROUGHOUT a program execution (so you can inspect the points in the code when max RAM is being used, and make sure the max is not being exceeded?)

If anybody has a link to some literature on how variables/data are treated (passed from flash to RAM, manipulated etc...) on a low down level, i'd really appreciate to read some more. It's hard to find some of the fundamentals well laid out on the web; still trying to wrap my head around the whole machine level process.
Thanks alot, -Nick

To lay out a few questions for succinctly:

1. when a variable is declared locally in a function, is it written from flash to RAM, used, then wiped out afterwards
2. do global variables reside in RAM whenever the program is running
3. all variables must be written in flash somewhere, right? RAM does not persist when the power is off, so I assume that at bootup variables are written from flash to RAM
4. When you use a function like Serial.print("Hello World"); is the string "Hello World" created as a global variable string that persists in RAM? Or is it created when the function is called, then wiped out of RAM after it is used?

nicholas.masson:
To lay out a few questions for succinctly:

1. when a variable is declared locally in a function, is it written from flash to RAM, used, then wiped out afterwards
2. do global variables reside in RAM whenever the program is running
3. all variables must be written in flash somewhere, right? RAM does not persist when the power is off, so I assume that at bootup variables are written from flash to RAM
4. When you use a function like Serial.print("Hello World"); is the string "Hello World" created as a global variable string that persists in RAM? Or is it created when the function is called, then wiped out of RAM after it is used?

1. No. It occupies space on the stack, which is in RAM, for the life of the function call.
2. Yes.
3. Yes.
4. The string "Hello World" is created as a global variable string that persists in RAM

Thank you! I didn't quite get your first response; you say that local variables persist

for the life of the function call

. Is that to say that the RAM to which local variables are stored is freed up when the function has completed (I assume so otherwise risk memory leaks...). If this is true, I would assume it also applies to the main 'void loop()'; for example:

int first_var;

void setup()
{
}

void loop()
{
function1();

function2();

Serial.print("Hello World");

function3();
}

Correct me if i'm wrong, but I believe the following is true:

1. loop begins, string "Hello World" does not exist in RAM memory
2. function1() and function2() are executed, "Hello World" still not occupying RAM
3. THEN the string HELLO WORLD get created and enters RAM
4. function3() executes, HELLO WORLD still in RAM
5. loop returns, all variables (including HELLO WORLD) wiped out, and function begins again

If this is not the case, and local variables exist after their parent functions finish, then perhaps my code would benefit from dynamic allocation and 'delete *pointer' to clear up memory that would be used by a function later down the line? I've never worried much about memory, but this seems like a good opportunity to learn about memory allocation and become a better programmer!

For your points 1 to 5 above: No.

Unless you move the string "Hello World" to progmem it will occupy RAM throughout the running of your sketch - it is not a local variable to loop, it is a constant string occupying precious space. Consider this though:

void loop()
{
char* str="Hello World";
int len=strlen(str);
Serial.println(str);
}

When loop is invoked, space is set aside on the stack for str and len; str is set to point to the already permanently allocated "Hello World". len takes the length of the string. When loop exits, the stack pointer moves back and len and str are gone. Next time loop runs, the stack pointer will again advance to make space for them. When main calls another function, its local variables will use the same space on the stack that those in loop used. So local variables go out of scope, but string constants are forever. Put them in progmem.

Why is it that locally declared strings are not wiped from RAM, whereas locally declared ints, etc... are cleared? (This also applies to arrays, right? so character arrays, i.e. strings are not cleared, whereas integer arrays locally declared are cleared after the function has exited?). Is this just a facet of the C programming language?

Thanks,
-Nick

A locally declared character array would not take up any memory after its containing function exited. In my example though, I've declared a pointer and pointed it to a constant string. That string data is inherently global. Every time loop runs, str will be pointed to that string - it has to exist in RAM.

Ah, okay... I see how the pointer will keep pointing to a permanent place in memory regardless of the local function etc...

Why then would the following string not be deleted when the void loop() exits and returns:

void loop()
{
Serial.print("Hello World");
}

It seems like from what you're saying (in previous comments), the Serial.print function will create a "hello world' string, and even after it has printed, and loop has exited and returned to the beginning, the "hello world" string is still in memory... If the program calls Serial.print("Hello World") a second time, will it create another "hello world" string in RAM, or go back to the first one, or is the string "hello world" indeed deleted and recreated in memory each iteration? I'm just confused about how the Serial.print() function operates when you give it a string directly. So would there be a difference between the above code, and the following code concerning how the 'string' array is saved (or deleted from memory).

void loop()
{
char to_print[]="Hello World";
Serial.print(to_print);
}

Is this code treated differently (in as far as memory) than if I pass "Hello World" directly to Serial.print()? Would this code clear the character array to_print[] from memory after the loop() function has finished executing?

Why then would the following string not be deleted when the void loop() exits and returns:

Would it make any difference if it was? The loop() function gets called again, and the string is needed again.

I'm just confused about how the Serial.print() function operates when you give it a string directly.

The compiler makes a pass through your code, and finds all constant strings, among other things, and places them at specific locations in SRAM. The call is then changed to print the string at that address. It does this for all constant strings, not just those used by Serial.print().

So would there be a difference between the above code, and the following code concerning how the 'string' array is saved (or deleted from memory).

No. The compiler is smart enough to do what that code sample does, for all strings. You can look at the assembler output for the two cases. I think that you'll find that they are very similar, if not identical. Different addresses, perhaps, in SRAM, but the same result - the Serial.print() function is called with the address where the string to print is stored.

Would this code clear the character array to_print[] from memory after the loop() function has finished executing?

No.

1. A totally different approach relates to the content of the log file.
   Have the log file be a data file, like xxx.CSV not a report, use comma instead of multiple blanks as delimiter, save space on SD card too.
   also no headings, keep time in unixtime only.

Write a simple program to display the data or used a spread sheet program.

2. Additionally, now you know where "ERROR CODE #nnnn" comes from, instead of strings.

3. Reduce memory requirements (and have better code too) move duplicate code to a function:
   i.e.
   for(int i=1;i<=10;i++)
   {
   digitalWrite(green_led, HIGH);
   digitalWrite(red_led,HIGH);
   delay(500);
   digitalWrite(green_led,LOW);
   digitalWrite(red_led,LOW);
   delay(500);
   }

+++++++++++++++++
Then there is the difference between
the code generated for: if(ECHO_DEBUG)
as opposed
#if ECHO_DEBUG
...
#endif

The #if...#endif either generates the code if ECHO_DEBUG is true otherwise doesn't generate code but
doesn't generate code to do the test which occurs at runtime.

It's not much, but sometimes every bbyte counts.
++++++++++
While we're on compiler directives ( #if, #endif) using that is a better approach than commenting out code like with calibrate. Defining a compile time variable to determine if the code is generated or not. This also reduces chances for errors and forgetting to comment out one of several sections.

Let us know how you did.

Great! Thanks for all the feedback. Keeping an eye on available memory is certainly a good motivator to code better. I've cleaned up alot of the slop and I can now run the code with a good bit of extra RAM (I used the memoryfree.h library for this). I didn't get around to doing it, but if I expand the code some more and memory once again becomes an issue, then I will probably store most of the long strings (like the file header, etc...) in the program memory, and just poll it when needed. using #if, #end if is certainly a good idea as well; no use in having the extra stuff floating around if ECHO_DEBUG is false...

Two questions still as to how I could do the following:
(1)

Defining a compile time variable to determine if the code is generated or not.

The arduino GUI will usually tell you when the code compiles incorrectly, but with my running-out-of-RAM problem, the GUI would say everything would compile correctly, but still not work. Is 'defining a compile time variable' a way to check for deeper errors?

(2)

You can look at the assembler output for the two cases.

Are there any good programs that will help to look at the assembler output/machine code, and show you where stuff is coming from/going? I can imagine such a tool would be very useful in understanding the guts of a compiler, or coding in general.

Thanks again for all the tips!
-Nick

Hi all

Am having a few problems with the SD Library. Have just added a #include at the top of an existing script and the complier takes a fit, doesnt like it at all. I am making a few changes to the rsteppercontoller script, i am adding an sd card function that i have been working on. Any ideas, i have tried compiling in an earlier version but no luck.
Clive

chlluk:
Have just added a #include at the top of an existing script and the complier takes a fit, doesnt like it at all.

You've done something wrong.

That's probably not much help to you, but I'm sure you can figure out why it isn't possible to provide a more helpful answer based on the information you've provided so far.