arduino2560  crashes after running 2-10 hours(SD)

Forum 2005-2010 (read only) Software Interfacing
1

Hello,

I've build the following powermonitor system/thermostat:

  1. an arduino MASTER with ethernet/sd shield (sdfat libs) + LCD display + temperature (ds1820) sensor + RF transeiver module (RFM12)
    It measures temperature and acts as thermostat (PID in software). It also receives temperatures and power measurement values from slaves and write all this data to SD card .
  2. jeenode slave(s) with temperature(DS1820) sensor (send value to master every 60 sec) and power monitorring (CT sensor,send value to master every 20 sec).

It all works well, but the arduino always crashes after 2-10 hours running. It's always hanging on this code (but it was already executed about 500-10000 times in these 2-10 hours running):

if (!file.open(&root, data_file, O_CREAT | O_APPEND | O_WRITE)) {sprintf(msg, "open failed data.csv file"); writeError(msg);}
file.print(currentDateTime);file.print(';');file.println(msg2);
if (file.writeError) {sprintf(msg,"write failed data.csv file"); writeError(msg);}
if (!file.close()) {sprintf(msg,"closed file failed data.csv "); writeError(msg);}

to be more specific: on the file.print or file.writeError it ends.

I've tried:

  • with less writes to SD card (with syncvalues every 10 min) and file.sync etc...
  • lower the speed of the spi bus
  • checked the available memory (always 2500 bytes SRAM available)

Does anyone know how to proceed (more debug) ?

Thanks, Jeroen.

2

It all works well, but the arduino always crashes after 2-10 hours running. It's always hanging on this code

Is it crashing (i.e. resetting) or is it hanging (not returning from a function that you think it should return from)? They are two very different things.

I'd start by commenting out the code that receives data from the other nodes. See if the problem is strictly related to the one Arduino with shield.

Are you using the ethernet capabilities, or just the SD card? If you are using the ethernet capability, why not have the server store the data, instead of the SD card?

3

Hello PaulS,

it is not crashing (resetting), but hanging.

When I comment out the code (see start reply) of the SD shield open/write/close then, this hang doen NOT occure.

From the SD/ethernet shield, I only use the sd part, the CS pin from the ethernet always stays high (not selected). The master is receiving data from the other nodes via a RFM12 module (self made soldered).

I'm now testing with less frequent writes to the SD (1 every 10 min instead of 6 writes per minute) and it seems promasing.
The fault did not occur (now 12 hours running)

4

Things usually hang in a loop when the program is waiting for something to happen, like a device to signal it is ready. Track down exactly where the hang is and put a time out on the amount of time you will wait for the condition to be right. Then you can take steps reinitialising things. The watch dog timer is also useful for doing this sort of thing.

5

Thanks Grumpy_Mike

never thought of WDT using here.

I've put in my setup() loop the 'wdt_enable(WDTO_8S);'
In my main loop the reset: ' wdt_reset();'

After a while, it hits the 8sec while writing to SD (real problem), but the WDT does not reset the arduino.
Am I missing here something ?

Thanks.

6

There was one topic that had some corrections to the SD card lib, as there where major flaws in it, one of the flaws was that the lib wants to fill the card with EOF(end of file), and that there wasnt a pointer storing the actual address of the last read/written data to the SD, so every time you send a write command the lib enters a gigantic for loop until it reaches the free space that is after the last written data, so it could start taking some HOURS to read or write something as the for loop takes more and more time to find where to write/read.

7

Hello Senso,
I'm using the latest sdlib.

  1. Can you remember were this issue for SD writes was reported.
  2. do you know what I'm doiing wrong with the WDT ?

Thanks,
Jeroen

8

Its a recent topic made here in the arduino forum, but I dont remenber the topic or the author name, sorry.

9

file.print(currentDateTime);file.print(';');file.println(msg2);
if (file.writeError) {sprintf(msg,"write failed data.csv file"); writeError(msg);}

You check file.writeError after three print statements; what if the error happens after the first or second print? Maybe build the complete string first before writing to the file.

Rob

10

Good seen Rob,

I've changed some things in writeSD function:
...
Serial.print("1: ");Serial.println(freeMemory());
snprintf(msg3,sizeof msg3,"%s;%s",currentDateTime,msg2);
if (debug) {Serial.print("Write to SD: "); Serial.println(msg3);}
file.println(msg3);
Serial.print("4: ");Serial.println(freeMemory());

if (file.writeError) {snprintf(msg3,sizeof msg3,"write failed data.csv file"); writeError(msg3);}

but it still hangs after logging on Serial:
1: 2221
Write to SD: 30/10/2010;22:27:51;Living ;T;S0;22.25;0;

So the code didn't get to the writeError part

11

Strange.
Just a thought: Did you try another SD card (other manufacturer)?
Could you run SdFatInfo.pde (in the examples of the sdfat.lib)

Rob

12

looked again in my code to see hints and should root not be used without & ??

if (!file.open(&root, data_file, O_CREAT | O_APPEND | O_WRITE)) 

===> 

if (!file.open(root, data_file, O_CREAT | O_APPEND | O_WRITE))

Can you show the complete code, or at least how the SDcard relevant variables are declared

my

// libraries for FAT
#include <SdFat.h>
#include <SdFatUtil.h>

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

void LogData()
{
  char buffer[128];
  
  sprintf(buffer,"meaningfull data");  // change at will
  
  int len = strlen(buffer);
  
  // write to FAT
  char name[24];
  sprintf(name, "%04d%02d%02d.log", now.year(), now.month(), now.day());

  // initialize the SD card
  if (!card.init()) Serial.print(card.errorCode(), HEX);

  // initialize a FAT volume
  if (!volume.init(card)) Serial.print(card.errorCode(), HEX);

  // open the root directory
  if (!root.openRoot(volume)) Serial.print(card.errorCode(), HEX);
  
  file.writeError = false;
  
  if (!file.open(root, name, O_CREAT | O_APPEND | O_WRITE))
  {
    Serial.println("error opening file");
  }
  file.println(buffer);

  file.timestamp(T_WRITE, now.year(), now.month(), now.day(), 
                              now.hour(), now.minute(), now.second());
  file.close();
 
}
13

Hello Rob,

my code:

#include <Sd2Card.h>
#include <Sd2PinMap.h>
#include <SdFat.h>
#include <SdFatmainpage.h>
#include <SdFatUtil.h>
#include <SdInfo.h>
...
// initialize SD card
Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

// store error strings in flash to save RAM for SD card
uint32_t syncTime = 0; // time of last sync()
#define SYNC_INTERVAL 600000 // mills between calls to sync()
#define error(s) error_P(PSTR(s))
const char data_file[] = "data.csv";
const char config_file[] = "config.txt";

void error_P(const char* str) {
PgmPrint("error: ");
SerialPrintln_P(str);
if (card.errorCode()) {
PgmPrint("SD error: ");
Serial.print(card.errorCode(), HEX);
Serial.print(',');
Serial.println(card.errorData(), HEX);
}
while(1);
}
...
in setup loop:
printLCD(1,0,"SDcard ");
device("SD");
snprintf(msg,sizeof msg, "card.init"); writeLog(msg);
if (!card.init(SPI_HALF_SPEED)) {snprintf(msg,sizeof msg,"ERROR SD card.init failed");writeError(msg);}
// initialize a FAT volume
snprintf(msg,sizeof msg, "volumeinit"); writeLog(msg);
if (!volume.init(&card)) {snprintf(msg,sizeof msg,"ERROR SD volume.init failed");writeError(msg);}
// open the root directory
if (!root.openRoot(&volume)) {snprintf(msg,sizeof msg,"ERROR SD openRoot failed");writeError(msg);}

...
the special function to log to SD (writeSD):
void writeSD(char* msg2)
{
device("SD");
getDateTime();

if (!file.open(&root, data_file, O_CREAT | O_APPEND | O_WRITE)) {snprintf(msg3,sizeof msg3, "open failed data.csv file"); writeError(msg3);}

Serial.print("1: ");Serial.println(freeMemory());
snprintf(msg3,sizeof msg3,"%s;%s",currentDateTime,msg2);
if (debug) {Serial.print("Write to SD: "); Serial.println(msg3);}
file.println(msg3);

if (file.writeError) {snprintf(msg3,sizeof msg3,"write failed data.csv file"); writeError(msg3);}
Serial.println(freeMemory());
if (!file.close()) {snprintf(msg3,sizeof msg3,"closed file failed data.csv "); writeError(msg3);}
Serial.print("after file.close freeMemory()=");
Serial.println(freeMemory());
}
..,
the initialisation of SD on each writeSD:
snprintf(msg2,sizeof msg2, "SD active"); writeLog(msg2);
digitalWrite (49, HIGH); //set RFM12 off
digitalWrite (10, HIGH); //set ethernet off
digitalWrite (4, LOW); //set SD off

If you need the full code, I just post it.
I will change the root part already.

Jeroen.

14

Did changing the "root" part solve the problem?
Be sure to download last sdfat lib Google Code Archive - Long-term storage for Google Code Project Hosting.

rob

15

Hello Rob,

I didnt' try root change yet, but I think I found the solution :

I'm discovered when I detach the RFM module (hardware wise) the problem does not occur.

So the SD logger is working as designed (perfect).

When I attach the RFM module, it always goes wrong after a while (now when writing every 2 sec values to SD and receiving every 10 sec from RFM module) approx. 1 hour.

So it sure has to do with the hardware.
I thing it has most probabily to do with a hardware interrupt (cominf from RFM12 module) during a SD write.

So I will test now with extra code before the sd write, to set the RFM_IRQ pin as output (to disable the IRQ signal)
and after the write set this back to input.

Fingers crossed...

16

Could you tell more about the RFM module ? datasheets? There may be a pin conflict?

17

Hello Rob,

it is just a hope RFM12 module, which uses the SPI bus, but I've changed the CS (instead of pin 53, I know use 49, bacause the sd/ethernet shield also uses pin 53 as CS for the ethernet device on the SPI bus)

And I did not had the problem, so I can be reasonably sure that disable the RFM IRQ before write to SD solved the problem.

Thanks,
Jeroen.

18

This?

19

Yes, this is the module.

And I can tell you for sure the problem is solved by disabling the RFM IRQ (setting the pin from arduino mega as output) before using (writing to) the SD card.

Best regards,
Jeroen.