Slow down in write times (with eventual crash) using TinyFat

As seems to often be the case, I'm implementing some data logging (monitoring the humidity in my damp cellar!).

I switched to TinyFat in the hope that it would better handle the SDCard being removed and re-inserted whilst the program was running than the standard SD library manages, and it does (I do realise this is risky and guaranteed to fail every now and again), however....

I open my logging file, write a line and close it after each write (unlike in the TinyFat example which writes 1Mb with only one file open and close operation). 30 seconds later I log another line. I can see on the Adafruit data logging shield that the access times get longer and longer (the red led which lights up on write), and after only about 30 lines the system hangs.

This isn't a free memory issue (I keep track of this, writing it to an LCD display and also in the log files themselves).

I don't really understand what I'm doing wrong. TinyFat doesn't offer a flush function, and I really don't want to leave the file open all the time (I want to be sure the data has been logged).

The writeLog function gets called in the main loop function, with a 30 second delay period...

I haven't found anything anywhere else about a similar issue with TinyFat, so I suspect it may be the way I am (abusing) the library? Does anybody with a bit of experience see where I'm going wrong? :blush:

//*** The main function controlling the logging

void writeLog(){
  if (openLogFile()){    
    logSensorData();
    closeLogFile();
  } 
  else {
    //Warn somebody?
  }
}

boolean openLogFile(){
  if (!file.exists(filename)){
      //If we can't find the file, it might be because the SDCard has been removed at some point. 
      //Lets try to reinitialize in case it has been reinserted - this works quite well
      initialiseSDCard();
      if (!file.exists(filename)){
        return false;
      }
  } 
  //Now lets open the file for appending
  res=file.openFile(filename, FILEMODE_TEXT_WRITE); 
  if (res!=NO_ERROR) {
    printLCDUserMessage(verboseError(res));  
    delay(2000);
    return false;
  } 
  return true;
}

void logSensorData(){

  strcpy(logfileLineStr,dateTimeCharArray);
  strcat(logfileLineStr,",");
  strcat(logfileLineStr,sensor1Str);
  strcat(logfileLineStr,",");

  .....

  file.writeLn(logfileLineStr);
  dataLogged = dataLogged + 1;
  char dataLoggedStr[8];
  sprintf(dataLoggedStr,"%02u",dataLogged);
  strcpy(userMessage,"Logged Data ");
  strncat(userMessage,dataLoggedStr,MAX_STR);
  printLCDUserMessage(userMessage);    
}


void closeLogFile(){
  file.closeFile();
}

Oh Bugger. I thought the forum post wasn't working (no success message, and it didn't appear in the list on the first couple of refreshes)... now I've ended up posting it three times - the forum won't let me delete my own messages!!! So sorry about that.

There's some sort of caching system in place, so give it a few minutes before reposting. I deleted your other posts.

Without actually reading your code, I can advise that I had a temperature and humidity sensor working, which writes to an SD card, with no problems.

http://www.gammon.com.au/forum/?id=12106

You may get some ideas from that.

There are similarities, and I'm impressed by the way you've gone down to such a low level.. I'm not there yet :-)

I'd like to stick with TinyFAT for now (not least because I find the GPL3 of the SD library is a step too far - IMHO).

I think, however, I may have posted prematurely. I'll need to dig in some more to get closer to the problem, although I've already lost too many evenings over the last couple of weeks trying to get reliable logging working (which is immune to the SDCard being removed and reinserted, the bigger issue). Oh, for a little more knowledge!!!

Many thanks!

monitoring the humidity in my damp cellar!

30 seconds later I log another line.

I don't know what process you are using to modify the humidity, but in a room of any real size, the humidity REALLY isn't going to change much in 30 seconds. Why are you logging data far faster than it can, in reality, change?

A toggle switch to define when it was safe to write (card is in place) and when it is not (card is removed) would seem to make sense to me.

The process of actually transferring data to the SD card involves buffering. There is no reason to store all the data in an array, and then write that array to the buffer. Skip all the strcpy() and strcat() stuff, and just write to the file (which really means write to the buffer).

Without seeing ALL of your code, we can not verify that you are not writing beyond the end of arrays, which can cause all kinds of mysterious behavior.

I think your function to write to the LCD could use some work, too. Instead of formulating the whole message, make the function know where on the LCD to write, and write the message in portions, using multiple calls, rather then possibly overwriting arrays.

Thanks for the response, I'll go through it more carefully when I have a moment...

As for the 30 seconds... my setup drives fans, a dehumidifier and a heater. It's quite dynamic, taking into account safe temperature thresholds for admitting air, also, while the devices are off, it tries to work out the rate at which humidity picks back up, so that in the long run we get some insight into the rate at which the cellar is drying out. These factors make the 30second resolution about right.