Sd Data Logging with Interrupts

Hi, I am using the Sd card along with DS1307 RTC to log some data from the sensors. I have 8 sensors and the way I am capturing the data is using timer interrupt every 0.5 sec. In the ISR, I read all the digital pins connected to the sensors and set a flag. In the loop I check for the flag and if it is set then I capture the time from RTC and log the sensor readings with time to the SD card. But the problem I am facing is that the arduino hangs after the first write to SD or sometimes it does not even writes to the SD and hangs. Is there a limitaion with the SDfat library or I2C library to not to use interrupts with it. I treid running the logging and reading of sensors by interrupt separately and they both work without any problem. Does anyone have any idea what the problem might be? or has anyone came across a similar situation like mine. Your help is much appreciated. Thank you

You need to provide more detailed information. Post your sketch.

Which pins are your sensors connected to. You must not connect sensors to the SPI pins or the SD chip select pin.

I am not using SPI pins for the sensors. In fact I am using a parallel to serial shift register to save the I/O pins for other purposes. following is my code:

// This sketch also reads the sensors
#include <SdFat.h>
#include <Wire.h>
#include <DS1307.h>
#include <SDLog.h>
#include <time_t.h>
#include <DataLog.h>
#include <Sensor.h>
#include <ShiftxRegister.h>

#define S_CLK 37
#define S_DATA 39
#define S_LOAD 36
#define S_CLKI 38 //optional clear pin

ShiftxRegister *reg; //instance to read the sensors connected to parallel to serial shift register
DataLog *logger; //data logger instance
Sensor *s0, *s1, *s2, *s3, *s4, *s5, *s6, *s7; //sensor class instance
boolean isr = false;

void setup() {
logger = DataLog::getInstance(46); //SD card CS pin is connected to Arduino Mega pin 46. Pin 53 is made output and is not in use by any other device
delay(100);
logger->sysLog(“Power ON”); //logs to system file using sdfat library
Serial.begin(115200);
if (!Serial) {
logger->sysLog(“Serial port initialization failed”);
}
s0 = new Sensor(); //these instances just generates name in the memory and has nothing to do with the logging
s1 = new Sensor();
s2 = new Sensor();
s3 = new Sensor();
s4 = new Sensor();
s5 = new Sensor();
s6 = new Sensor();
s7 = new Sensor();
reg = new ShiftxRegister(S_CLK, S_CLKI, S_LOAD, S_DATA);
delay(1000);

cli();//stop interrupts
//set timer1 interrupt at 1Hz
TCCR1A = 0;// set entire TCCR1A register to 0
TCCR1B = 0;// same for TCCR1B
TCNT1 = 0;//initialize counter value to 0
// set compare match register for 1hz increments
OCR1A = 15624;// = (1610^6) / (11024) - 1 (must be <65536)
// turn on CTC mode
TCCR1B |= (1 << WGM12);
// Set CS12 and CS10 bits for 1024 prescaler
TCCR1B |= (1 << CS12) | (1 << CS10);
// enable timer compare interrupt
TIMSK1 |= (1 << OCIE1A);
sei();//allow interrupts
}

void loop() {
if (isr) {
cli();
readSensors();
isr = false;
sei();
}

}

ISR(TIMER1_COMPA_vect){//timer1 interrupt 1Hz toggles pin 13 (LED)
//generates pulse wave of frequency 1Hz/2 = 0.5kHz (takes two cycles for full wave- toggle high then toggle low)
isr = true;
}
void readSensors() {
cli();
int data = reg->getData(); // getData() reads the data from Parallel to Serail shift register and returns a int
logger->sysLog((String)data); //write to system file using SDFAT library
sei();
}

SdFat was not designed to run with interrupts disabled. It uses millis() to check for timeouts.

An SD write can take many milliseconds. Occasional write times of 100 milliseconds are not uncommon. The SD spec allows an SD write to take 250 ms. Many timer0 interrupts will be missed.

Wire also uses interrupts so it will hang if you read the RTC with interrupts disabled.

You have a strange nesting of disable/enable interrupts in loop() and then in readSensors().

I see you use "new" to allocate memory. Do you use "new" in any of your libraries? I cringe when I see dynamic memory used after start-up in embedded systems.

Edit: I see you use String so you do use dynamic memory in readSensors() with interrupts disabled.

I am disabling and enabling the interrupts to make sure the writing to SD does not get disturbed. But as you are saying that sdfat is not designed to work with interrupts disabled. And further when reading the RTC, interrupts should be enabled. Perhaps, disabling the interrupts is causing the trouble. What I am trying to achieve is to read my sensors every 0.5 seconds, get the time and date from RTC and log it to SD card. Is there a better approach to this problem?. The answer to why I am using new to allocate memory is because during the sensor class instance, some of the information is read from the SD card. And to make sure there is no more than one instance of SD2card, I created a singleton Datalog class which is shared across all the classes in the project. And to make all this to work, I beleive (might be wrong) I cannot create instances of sensor class outside the setup or loop. Therefore I create pointers of these class and then point them to instances of sensor class in the setup. I am not using new() anywhere else in my code. If there is a better approach to it, I will be more than thankful to have your answer.

This is a simple slow data logging problem. You don’t need a timer and ISR.

Do something like this:

const uint32_t READ_INTERVAL = 500;  // Read every 0.5 seconds.

uint32_t readTime;

void setup() {
  Serial.begin(9600);
  readTime = millis();
}
void loop() {
  while ((millis() - readTime) < READ_INTERVAL) {}
  readTime += READ_INTERVAL;
  
  // Replace print with code to read and log data.
  Serial.println(readTime);
}

I can’t comment on your use of new() since I have no idea what is in your classes.

C++ is used in may embedded systems but it is common for projects to forbid use of dynamic memory or limit use to program start-up.

Why do you worry about The Sd2Card class? Do you use the SdFat class?

Why not just use instances at global scope and avoid malloc/new?

I will agree with the while loop technique but I have other things as well in the loop and to make sure I am reading my sensors every 0.5 or 1 second, I dont see any other technique beside interrupts. I commented cli, sei and the program does not hang anymore. The write time to sd card for this particular data is 40 ms which I believe will not be a problem. Thank you for the help. Coming to the point of global instances; several classes in my code are logging/reading data to/from the SD card. Therefore, I created one instance of the Datalog class and wherever I need to use I just use a pointer to point to that instance. Following is my sensor class constructor:

Sensor::Sensor() { id = unique_id++; name = logger->getSensorName(id); location = logger->getSensorLocation(id); logger->sysLog(("Sensor Initialized, id : " + (String) id) + "; name : " + name); }

if I declare global variables for my sensor i.e. Sensor s1, s2, s3; before the setup, my arduino hangs..

I would love to see what your comments are on this..

I will agree with the while loop technique but I have other things as well in the loop and to make sure I am reading my sensors every 0.5 or 1 second, I dont see any other technique beside interrupts.

Your flag is useless for accurate read timing. You can change the while loop to do other thing but there will be jitter in read time.

void loop() {
  if ((millis() - readTime) >= READ_INTERVAL) {
    readTime += READ_INTERVAL;
  
    // Replace print with code to read and log data.
    Serial.println(readTime);
  }

  // Do other stuff.
}

if I declare global variables for my sensor i.e. Sensor s1, s2, s3; before the setup, my arduino hangs..

This is because the constructor gets called before setup. If a class is used at global scope it needs a simple constructor. In general constructors used in embedded systems should not do stuff like you are doing for two reasons. The class can not be used at global scope. If the constructor fails, there is no clean way to indicate failure. Most small embedded systems don't support/use exceptions.

 logger->sysLog(("Sensor Initialized, id : " + (String) id) + "; name : " + name);

Using String is asking for trouble, String calls malloc(). Use of String would be forbidden in any serious embedded project.

For example: MISRA C stands for "Motor Industry Software Reliability Association" C standards.

MISRA-C++ Rule 18-4-1: Dynamic heap memory allocation shall not be used.

I commented cli, sei and the program does not hang anymore. The write time to sd card for this particular data is 40 ms which I believe will not be a problem.

Sometimes your write will take much longer than 40 ms. The SD card occasionally pauses to erase large blocks of flash and remap/copy areas for wear leveling.

I dont see any other technique beside interrupts.

You could use an RTOS. Here are small systems I ported to Arduino https://code.google.com/p/rtoslibs/.

Look at examples in NilRTOS20130720.zip. The nilSdLogger example can log data to an SD card at 100-1000 samples per second. Read jitter time is very low in this example. There is no point in reading data at high rates if you don't know when the data is read.

Other examples demonstrate three or more threads.

You will need to learn how to correctly use a small priority based preemptive scheduler.