Problem writing to SD card

I hope this is the correct place to ask this question. I think I have a code problem, rather than a hardware problem.
I am trying to write the input of an analog pin to a .csv file, once per second.
I am using an Arduino Uno with the Wireless SD shield, writing to a 1GB micro SD card formatted to FAT16.

The problem is that it writes several files, and then just seems to stop. The number of files it writes each time is not consistent ... so it is difficult to diagnose the exact problem.

I am quite new to coding, and to the Aduino in particular so it is most likely that I have made an obvious error. I have spent many days going over it and I can not think of any other way to fix it. Perhaps a more experienced person can spot my mistake?

The code is intended to:
Create a temporary string composed of 5 readings from the analog 0 pin, each spaced 1 second apart, each separated by line feed.
(Add a timestamp to each line if the RTC is attached).
Write that string to two files; "mainfile" and another file with a dynamically assigned file name.
Reset the string to empty, and repeat.

It is taking the voltage reading of a GSR sensor which is plugged into analog 0. The project is intended to allow the Arduino to be powered from a battery and worn all day while readings are taken every second and recorded to SD card. It is writing to two files because sometimes the files seem to get corrupted if power is removed during the write process. The "mainfile" is a continuous record of the data. In case this single massive file gets corrupted, the series of backup files contain the same data spread over many files. The series of backup files are called "split file" in the code. Currently each file is composed of only one file write (but this can be adjusted), and then a new file is created, and so on. (At the moment each file consists of only 5 seconds worth of data, but that will be adjusted to about 5 mins when it is working properly). The file name for each "split file" is composed of a single letter followed by a 7 digit number, and each file is named sequentially, for example, "A0000001", "A0000002", "A0000003", etc.

It seems to be correctly writing data to the file for the first few rounds, but eventually it stops. It seems to stop most often at split file number 9 or 26.

In addition it seems to go wrong when I plug the RTC in. I DID have the RTC working correctly, but now when I plug it in, the data which is written to the file just comes out as random characters (squares, currency signs, symbols, etc.). I had no idea why it is doing that, because I can't remember changing anything to cause it to happen!

Can anyone see if there is something majorly wrong that I am doing in the code?

(I will post the code in the next post, because I have run out of characters here).

/*
 On the Ethernet Shield, CS is pin 4. Note that even if it's not
 used as the CS pin, the hardware CS pin (10 on most Arduino boards,
 53 on the Mega) must be left as an output or the SD library
 functions will not work.     
 */

#include <SD.h>
#include <Wire.h>
#include "RTClib.h"
RTC_DS1307 RTC;

// External Variables ----
int LinesPerString = 5;     // LinesPerString = 10 & WritesPerFile = 30 = 1 file per 5 minutes
int WritesPerFile = 1;
boolean UseSerialPrint = true;
boolean UseErrorBlink = false;
boolean UseTimestamp = true;
boolean UseMainFile = true;
boolean UseSplitFile = true;
char FileNamePrefix = 'A';
const int chipSelect = 4;    // Must be set to 4 as default for SD card library
int led = 7;                 // pin for LED
int baud = 9600;

// Internal Variables ----
int millicount;
int RTC_OK = 0;
int SD_OK = 0;
int Serial_OK = 0;
int WritesPerFileCounter = 1;
int LinesPerStringCounter = 1;
String dataString = "";
String timestamp;
int LoopOK = 1;
char filename1 = 48;
char filename2 = 48;
char filename3 = 48;
char filename4 = 48;
char filename5 = 48;
char filename6 = 48;
char filename7 = 48;
char MainFileName[9] = { 'm','a','i','n','f','i','l','e' };  
int sensor;
//File dataFile;
//File dataFileMain;
  
void setup() // ----------------------------------------------------------------------------------------
{
// ----
 
  // Initialize the digital pin as output
  if ( UseErrorBlink == true ) 
     { 
     pinMode(led, OUTPUT); 
     }
    
  // RTC stuff 
  Wire.begin();
  RTC.begin();
  
  // Initialize default CS pin
  pinMode(10, OUTPUT); // make sure that the default chip select pin is set to output, even if you don't use it  
  //pinMode(chipSelect, OUTPUT);
     
  // Open serial communications 
  Serial.begin(baud);
  if ( Serial )
     { 
     Serial_OK = 1; 
     SERIAL_PRINT("Initializing SD card..."); 
     }

  // Check RTC ----
  if ( RTC.isrunning() ) 
     {
     SERIAL_PRINT("RTC is running.");
     RTC_OK = 1;
     }
  else
     {
     SERIAL_PRINT("ERROR: RTC is NOT running!"); 
     }
 
  // checkl SD ----
  if (SD.begin(chipSelect)) 
     {
     SERIAL_PRINT("SD card initialized.");
     SD_OK = 1;    
     }
  else
     {
      SERIAL_PRINT("ERROR: SD Card failed, or not present");
     }

// ----  
}
// -----------------------------------------------------------------------------------------------------

void loop() // -----------------------------------------------------------------------------------------
{
// ----  

if ( (LoopOK == 1) /*&& (RTC_OK == 1) && (SD_OK == 1)*/ )
{

SERIAL_PRINT("LoopOK = 1");
  
filename1 = 48;
while ( filename1 <= 57 )
{
filename2 = 48;
while ( filename2 <= 57 )
{
filename3 = 48;
while ( filename3 <= 57 )
{
filename4 = 48;
while ( filename4 <= 57 )
{
filename5 = 48;
while ( filename5 <= 57 )
{ 
filename6 = 48;
while ( filename6 <= 57 )
{
filename7 = 48;
while ( filename7 <= 57 )
{
WritesPerFileCounter = 1;
while ( WritesPerFileCounter <= WritesPerFile )
{

  // FILENAME ----
  char Filename[9] = { FileNamePrefix,filename1, filename2, filename3, filename4, filename5, filename6, filename7 };

  // DATASTRING ----
  LinesPerStringCounter = 1;
  dataString = "";
  while ( LinesPerStringCounter <= LinesPerString )
        {
        sensor = analogRead(0);    
        if ( (UseTimestamp == true) && (RTC_OK == 1) ) 
           { 
           timestamp = "";
           DateTime now = RTC.now();
           timestamp = String(now.year())+"."+String(now.month())+"."+String(now.day())+" "+String(now.hour())+":"+String(now.minute())+":"+String(now.second()); 
           dataString += "\n"+String(sensor)+"\t"+String(timestamp); 
           SERIAL_PRINT(String(sensor)+" "+String(timestamp));  
           }
        if ( (UseTimestamp == false) || (RTC_OK == 0) ) 
           { 
           dataString += "\n"+String(sensor); 
           SERIAL_PRINT(String(sensor)); 
           }
        delay(1000);
        LinesPerStringCounter++;
         }

  // WRITE TO SPLIT FILE ----
  if ( (UseSplitFile == true) && (SD_OK == 1) )
     {
     File dataFile = SD.open(Filename, FILE_WRITE);
     if (dataFile) 
        {
        if ( dataFile.println(dataString) <= 0 )
           { SERIAL_PRINT("error writing split file"); }
        dataFile.close();
        SERIAL_PRINT(dataString);
        SERIAL_PRINT(Filename);
        }  
     else 
        {
        SERIAL_PRINT("error opening split file");
        if ( UseErrorBlink == true )
          {
          ERROR_BLINK_CYCLE(5, 1000);
          digitalWrite(led, HIGH);
          }
        }
     }
 
  // WRITE TO MAIN FILE ----
  if ( (UseMainFile == true) && (SD_OK == 1) )
     {   
     File dataFileMain = SD.open(MainFileName, FILE_WRITE);
     if (dataFileMain) 
        {
        if ( dataFileMain.println(dataString) <= 0 )
           { SERIAL_PRINT("error writing main file"); }
        dataFileMain.close();
        SERIAL_PRINT(dataString);
        SERIAL_PRINT(MainFileName);
        }  
     else 
        {
        SERIAL_PRINT("error opening main file");
        if ( UseErrorBlink == true )
          {
          ERROR_BLINK_CYCLE(5, 1000);
          digitalWrite(led, HIGH);
          }
        }
     }
     
delay(1000);
WritesPerFileCounter++;
}
filename7++;
}
filename6++;
}
filename5++;
}
filename4++;
}
filename3++;
}
filename2++;
}
filename1++;
}

LoopOK = 0;
}

// ----
}

// FUNCTIONS -------------------------------------------------------------------------------------------------------------------------------------------------------------
// -----------------------------------------------------------------------------------------------------------------------------------------------------------------------

// ERROR_BLINK() ---------------------------------------------------------------------------------------------------------------------------------------------------------
void ERROR_BLINK_CYCLE(int cycles, int milliseconds)
{
if ( UseErrorBlink == true )
   {
   SERIAL_PRINT("ERROR_BLINK_CYCLE() function intiated");
   int counter = 0;

   while ( counter <= cycles )
         {
         digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
         SERIAL_PRINT("on");
         delay(milliseconds);               // wait for a second
         digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
         if ( Serial_OK == 1) {Serial.println("off");}
         delay(milliseconds);               // wait for a second
         counter++; 
         }
      
   digitalWrite(led, HIGH);
   SERIAL_PRINT("on");
   SERIAL_PRINT("ERROR_BLINK_CYCLE() function completed");
   }
return;   
}

// SERIAL_PRINT() -------------------------------------------------------------------------------------------------------------------------------------------------------
void SERIAL_PRINT(String printstring)
{
if ( UseSerialPrint == true )
   {
   if ( Serial_OK == 1) {Serial.println(printstring);}    
   }  
return;
}

The first thing to do is get rid of String and use C-style null terminated character arrays instead.
There is a known problem with the String library, on top of which there's only 2kB of Ram in total which is not enough to support the sort of memory usage that the String library requires.
You are also doing something really strange with nested while loops but fix the strings first.

Pete

OK great, thanks. I guess that will probably make a big impact. I'll change the strings and see what happens. Might take me a while to figure out how to do that :wink:

The nested loops are there for the file name of the "split file". According to the sd library I'm using I think the file name has to be char. I am creating the file name one character at a time. Each new file name has to be "the previous file name +1". The only way I could figure out how to do that was to increase the ASCII character value by 1 each time, for each digit, in the correct order. Each digit starts at 48 (ASCII for "0"), and goes up in 1's to 57 (ASCII for 9), then resets to 48 and the one to the left goes up by one - going from right to left. So the first file is called "A0000001", the second is called "A0000002", the third is called "A0000003", and so on until I run out of battery. Hence the nested loops. (Using while rather than for, so that it is slightly easier to see where the loops end). I'm not used to using arrays and chars to make strings. The programming language I normally use just has string, double, bool and int, with automatic implicit conversion between them. There is probably a better way to increase the file name by 1 each time!

Have you tried printing out your filenames to the serial monitor to see if it's doing what you want?

It would be much easier to use an integer variable for the file number and then use sprintf to convert that to the filename.
similar to:

long file_number = 0;
char file_name[16];
.
.
.
  file_number++;
  sprintf(file_name,"A%07d.txt",file_number);
  File dataFile = SD.open(file_name, FILE_WRITE);
etc.

Pete

Yes, I have printed the file names to serial, and they seem to be working fine.

I tried using an integer, and adding that to the string, to start with. That is the method I would normally have used. But it kept giving datatype mismatch errors. Hopefully sprintf will remove the need for the nested loops, which may well help prevent other problems from occurring as well. Thank you!

Hi,
Excuse me for cutting in, but I'm having exactly the same problem printing to the SD card on the Arduino wireless SD shield. I'm only printing data, not filenames, but I only get anywhere from 2 to 11 data points, while all the data show on the Serial monitor and X-CTU terminal. Also, the times are not incremented on the card, but only repeat the initial time. I'll post the problem under Project Guidance, adding to what I've already posted there.

I am using the UNO, but the only chipSelect that worked was 4, not 10. Is this true for you?

Regards,
Oldguy

I have 4 set as chipSelect, but am also setting both 4 and 10 to OUTPUT in the setup function (contrary to my code posted above). It may not be technically correct but it seems to work, and does not seem to cause any other problems.

I have implemented sprintf, and that seems to have fixed the problem! Thank you el_supremo.

These pages also helped:
http://arduino.cc/en/Reference/String
http://php.net/manual/en/function.sprintf.php

Just implementing sprintf for the data that is written to the SD card was not sufficient. If there are any other strings at all in the code it seems to clog the whole thing up.
I removed all strings and any references to them, and replaced them all with either char[] or text written directly into the .print or .println function.

I also removed the nested loops for generating the file name, and replaced them with an integer++. (I did not know about the "%07d" ability to place prefix zeros in this language).

The only problem now is that it writes to the SD card every second, rather than saving several seconds up and writing them all at once. This is annoying because it takes a second to write the data, so during that second no data is being collected. Also, it must reduce battery life. I will have to figure out how to join char arrays dynamically in a loop - it seems more complicated than just joining strings. I am assuming there will be some limit on the number of readings that can be joined together before the memory is overloaded and the original problem starts all over again.

The new (fixed and working) code is below:

/*

 On the Ethernet Shield, CS is pin 4. Note that even if it's not
 used as the CS pin, the hardware CS pin (10 on most Arduino boards,
 53 on the Mega) must be left as an output or the SD library
 functions will not work.     
     
 */

#include <SD.h>
#include <Wire.h>
#include "RTClib.h"

RTC_DS1307 RTC;

// External Variables ----

int LinesPerString = 1;     // LinesPerString = 10 & WritesPerFile = 30 = 1 file per 5 minutes
int WritesPerFile = 3;
int MaxFiles = 10000;

boolean UseSerialPrint = true;
boolean UseTimestamp = true;
boolean UseMainFile = true;
boolean UseSplitFile = true;

const int chipSelect = 4;    // Must be set to 4 as default for SD card library
int led = 7;                 // pin for LED
int baud = 9600;

// Internal Variables ----

int RTC_OK = 0;
int SD_OK = 0;
int Serial_OK = 0;
int WritesPerFileCounter = 1;
int LinesPerStringCounter = 1;
int LoopOK = 1;
char MainFileName[9] = { 'm','a','i','n','f','i','l','e' };  
int sensor;
int FilesCounter;

  
void setup() // ----------------------------------------------------------------------------------------
{
// ----
 
  // RTC stuff 
  Wire.begin();
  RTC.begin();
  
  // Initialize default CS pin
  pinMode(10, OUTPUT); // make sure that the default chip select pin is set to output, even if you don't use it  
  pinMode(chipSelect, OUTPUT);
     
  // Open serial communications 
  Serial.begin(baud);
  if ( Serial )
     { 
     Serial_OK = 1; 
     if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("Initializing SD card...");}
     }

  // Check RTC ----
  if ( RTC.isrunning() ) 
     {    
     RTC_OK = 1;
     if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("RTC is running.");}
     }
  else
     {
     if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("ERROR: RTC is NOT running!");}
     }
 
  // checkl SD ----
  if (SD.begin(chipSelect)) 
     {
     SD_OK = 1;    
     if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("SD card initialized.");}
     }
  else
     {
     if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("ERROR: SD Card failed, or not present.");}
     }

// ----  
}
// -----------------------------------------------------------------------------------------------------

void loop() // -----------------------------------------------------------------------------------------
{
// ----  

FilesCounter = 1;
while ( FilesCounter <= MaxFiles )
{

WritesPerFileCounter = 1;
while ( WritesPerFileCounter <= WritesPerFile )
{

  // DATASTRING ----
  LinesPerStringCounter = 1;
  char dataStringchar[25];
  while ( LinesPerStringCounter <= LinesPerString )
        {
        sensor = analogRead(0);    
        if ( (UseTimestamp == true) && (RTC_OK == 1) ) 
           { 
           DateTime now = RTC.now();
           sprintf(dataStringchar,"%d\t%d.%02d.%02d\t%02d:%02d:%02d",sensor,now.year(),now.month(),now.day(),now.hour(),now.minute(),now.second());
           Serial.println(dataStringchar);
           }
        if ( (UseTimestamp == false) || (RTC_OK == 0) ) 
           { 
           sprintf(dataStringchar,"%d",sensor);
           Serial.println(dataStringchar);
           }
        delay(1000);
        LinesPerStringCounter++;
         }

  // FILENAME ----
  char Filename[9];
  sprintf(Filename,"A%07d",FilesCounter);

  // WRITE TO SPLIT FILE ----
  if ( (UseSplitFile == true) && (SD_OK == 1) )
     {
     File dataFile = SD.open(Filename, FILE_WRITE);
     if (dataFile) 
        {
        dataFile.write(dataStringchar);  
        dataFile.println();  
        dataFile.close();
        Serial.println(Filename);
        }  
     else 
        {
        if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("ERROR: Problem opening split file.");}
        }
     }
 
  // WRITE TO MAIN FILE ----
  if ( (UseMainFile == true) && (SD_OK == 1) )
     {   
     File dataFileMain = SD.open(MainFileName, FILE_WRITE);
     if (dataFileMain) 
        {
        dataFileMain.write(dataStringchar);
        dataFileMain.println();
        dataFileMain.close();
        Serial.println(MainFileName);
        }  
     else 
        {
        if ((UseSerialPrint==true)&&(Serial_OK==1)){Serial.println("ERROR: Problem opening main file.");}
        }
     }
     
delay(1000);
WritesPerFileCounter++;
}

FilesCounter++;
}

// ----
}

I suppose I should also mention that I am not ending my file names with .txt or .csv

If they don't have any file endings then when you go to open them windows asks you to select which program you want to open them with. If you open them with Excel then it immediately brings up a warning message claiming the data is corrupt or in the wrong format (or something like that). If you just click to continue through to opening the file then it still opens the file correctly and all the data is still fine ... I did this because with .txt at the end of the file was causing the file to be completely unreadable, even though it was correctly being recognised as a txt file in windows (also it was printing funnily over the serial as well).

The reason Excel doesn't like your file is probably because you don't separate the fields with commas. It should work if you write the file with the .CSV extension and change the output format to this:

"%d,%d/%02d/%02d,%02d:%02d:%02d"

This will create three fields: sensor, YY/MM/DD, hh:mm:ss
The file should then automatically open with Excel when you double click the name in explorer.

Pete

OK, that seems to work better now. I didn't know that a comma worked in that way.
Thanks.