Datalogging with RTC

I started with the temperature/humidity weatherstation and added an SD card to sense the data every 1000 ms and record the actual temp, average temp, and humidity. Now I want to incorporate a RTC for more accuracy, but after reading about the numerous issues of the DS1307 regarding portability, different libraries etc im starting to rethink.

I don’t care so much about the time, as much as I care about accurate 1 second intervals over at least a 24 hour period. It would be nice to log the time/date at the start of each day, though.

Any advice on which library to use? I guess im also asking how to use the RTC for one second intervals instead of using delay(1000)?

Also how accurate is the arduino by itself? If I use just delay(1000) in a loop as I am now, how accurate will this be in 24 hours? If this is reasonably accurate I would be willing to abandon the actual time keeping as the libraries I’ve seen so far have been rather daunting for a newbie. I really just want to see how the temperature changes throughout the day.

An Arduino's clock is usually around ±0.5%. There really isn't a lot wrong with the DS1307 (what were the issues you refer to?), its accuracy depends on the crystal used, but is usually about 20-30 ppm. For a little more money, RTCs that are in the range of 2-3 ppm are common.

If the idea is to sample once a second, delay(1000) isn't the most accurate way to do it, because it takes time for the code that does the logging to run. So the actual interval will be 1000 ms plus the time to execute one logging cycle. Better to use millis() to know when 1000 ms has elapsed since the last sample. See the "blink without delay" example in the IDE if you're not familiar with this approach.

Just out of curiosity, how much is the temperature or humidity expected to change in one second? Some sensors can take that long or longer to take a reading and present the data. Sampling every second seems like overkill for typical environmental monitoring.

I submit that, if you are planning to monitor the weather at 1 second intervals, you are just kidding yourself. It's a dog chasing car situation, even to the point that relevant data could be drowned in the detritus. I'm no great meteorologist but several years of living on planet Earth rather suggests that reading at 60 second intervals would surely be as frequent as anybody would ever need, and would work wonders when it comes to planning any storage.

The DS1307 is fine, there are no portability problems and, while different libraries have different uses, you don't need any library to actually tell the time. I use RTCLib but it is only for creating the filenames. There is a library pair Time and TimeAlarms that you may find useful for regulating your sampling. I'm not sure the DS1307 is any more accurate than using an internal clock based on millis but it has the clear advantage of keeping time in the event of power/interent/GPS failure, and just seems the obvious choice.

The DS1307 can be corrected off a PC or the internet but I have yet to find a real need for this.

Using delay can't be that good an idea but I find it entirely satisfactory. The loop can be tuned to run round at essentially 1Hz. There may be the odd hour a when reading in seconds is missing, or extra, but I only record each tenth runround anyway - i.e. about six times more frequently than you need. Indeed, if you really just want to see how the temperature changes throughout the day, you don't need a DS1307 but they are so cheap, and useful, that they are pretty hard to pass up.

If you decide that you want a more stable RTC then take a look at the DS3231 etc. Temperature compensated, internal crystal… Can be read and written to using the DS1307 libraries, or write your own functions. The registers are fully explained in the data sheet.

I am using it as an indoor thermostat, it operates a fan heater and a cooling device or fan. The problem is if the sensor is in line with the heater’s air current then it changes too rapidly, and longer sampling intervals compounds the problem. The averaging takes care of this problem I think, since it allows the actual temperature to reach into the 30’s but only when the average temperature reaches the desired 24 degrees

Firstly, I think a one second sample rate is pretty lenient… I see people trying to read and write millisecond intervals so don’t tell me its too frequent. If however you are suggesting it not necessary, or conflicts with the RTC or DHT’s accuracy then I could reconsider.

As for the delay, I thought that the delays due to the sensor specs etc were defined in the code since you have to wait for the sensor to be ready using delay() etc. I didn’t realise there were additional hidden delays also.

The problems I was referring to I’ve heard “There isnt a standard library for the DS1307 for arduino”, which leads to a bunch of people writing libraries that arent complete, or they don’t have the right registry files, are using wrong functions which may conflict with other things etc all of which I have very limited understanding of at this stage.

and some DS1307 modules short at the battery and thus don’t hold the time when not powered. Thanks for the info on the alarms ill have a look.

DHT22 only updates every two seconds, DHT11 is one second IIRC also two seconds. I think reading them more often is OK but they will just return the same value. I've only played with DHT22, and just a little. A DS18B20 needs 750ms to do its A/D conversion at full resolution. So yes I'm afraid I've got bad news for anyone trying to read these sensors at millisecond intervals.

I do like a moving average approach, usually between 5 and 10 samples. I could support that, combined with a one second interval (or whatever the minimum for the particular sensor may be.) The other thing that's important on top of that is to have some hysteresis. I imagine that nearly every thermostat ever fielded has some built in, I want to say between 0.5°F and 2-3°F but that's just a SWAG. Any HVAC experts please feel free to chime in. I think some thermostats may have adjustable hysteresis. Sounds like that's what you need, luckily it's a snap to implement in software!

Certainly there are a lot of DS1307 libraries. Not sure any of them are "standard" whatever that may mean in this context. I've always used the one that comes with the Time library, it's very basic, doesn't implement all the RTC functionality but I haven't needed anything else. OTOH, it's very straightforward to read and write the registers with the Wire library to access all the functionality of the RTC.

Edit: DHT11 sample period is 2s per the datasheet.

Basically I am using the averaging as hysterisis, because by the time the actual readings come down they are close to the average anyway, but on the rise there are large temperature spikes, the averaging flattens out the pattern so the heater and fan constantly switching on and off. For example 24 degrees, I have if the actual temperature is smaller than 22 degrees... switch on heater, switch off when average is above 24 degrees. If the average temperature is higher than 26 degrees switch on a fan. So there is some hysterisis there also.

I think using the average to turn the heater off, and the actual to turn it on would be the best bet. Since it heats up alot faster than it cools.

I've actually thought about eventually extending it to have a more variable hysterisis involving heat equations, recording change in temperature. But for now I just need to know how to get the RTC to create regular intervals.

Regarding the RTC, I would either use it in conjunction with the Time library (the time library will sync its time to that of the RTC), or connect the RTC square wave output to generate an interrupt every second. Of course both could be done. By using the Time library, the code could be aware of not only time of day, but day of week, etc. A thermostat program could then have different settings by day or whatever if that’s important to your application.

I’d probably try both the moving average and hysteresis. The moving average reduces the short-term variations in the measurement, and the hysteresis prevents constant/short cycles of the heating or cooling equipment. Two different things, really.

So if I dont use the square wave interrupt (don’t have many digital pins left, unless you can use analog?), what can I write in the loop to make sure its only logging every second due to the RTC?

Something like:

if abs(second()-tempsecond)>0 {
write the log;
}
tempsecond=second();

Its all I can think of right now. How do you implement the square wave? I guess you just plug it into a digital pin and when it reads high you do whats necessary, but then how do you make sure the arduino only reads it once per second?

syphex:
So if I dont use the square wave interrupt (don’t have many digital pins left, unless you can use analog?), what can I write in the loop to make sure its only logging every second due to the RTC?

Something like:

if abs(second()-tempsecond)>0 {
write the log;
}
tempsecond=second();

Yes, something along those lines. (Yes the analog pins can function as digital pins as well.) Here’s a little more detail, this is the type of thing I usually do:

//Disclaimer: Compiles OK but not tested.

#include <DS1307RTC.h>     //http://www.arduino.cc/playground/Code/Time
#include <Streaming.h>     //http://arduiniana.org/libraries/streaming/
#include <Time.h>          //http://www.arduino.cc/playground/Code/Time
#include <Wire.h>          //http://arduino.cc/en/Reference/Wire

#define BAUD_RATE 115200

time_t utc;                //current time from the RTC (I almost always set my RTCs to UTC)
time_t lastUTC;            //last time one-second processing code was executed

void setup(void)
{
    Serial.begin(BAUD_RATE);

    //set up RTC synchronization
    setSyncProvider(RTC.get);
    Serial << F("RTC SYNC");
    if (timeStatus() != timeSet) {
        Serial << F(" FAIL");
        while (1);
    }
    Serial << endl;
}

void loop(void)
{
    utc = now();
    if (utc != lastUTC) {
        //once-per-second processing
        //...guaranteed only if the code here takes less than one second to execute!
        lastUTC = utc;
        printTime(utc);
        Serial << F("UTC ");
        printDate(utc);
        Serial << endl;
    }        
}

//print time to Serial
void printTime(time_t t)
{
    printI00(hour(t), ':');
    printI00(minute(t), ':');
    printI00(second(t), ' ');
}

//print date to Serial
void printDate(time_t t)
{
    Serial << dayShortStr(weekday(t)) << ' ';
    printI00(day(t), ' ');
    Serial << monthShortStr(month(t)) << ' ' << _DEC(year(t));
}

//Print an integer in "00" format (with leading zero),
//followed by a delimiter character to Serial.
//Input value assumed to be between 0 and 99.
void printI00(int val, char delim)
{
    if (val < 10) Serial << '0';
    Serial << _DEC(val) << delim;
    return;
}

Its all I can think of right now. How do you implement the square wave? I guess you just plug it into a digital pin and when it reads high you do whats necessary, but then how do you make sure the arduino only reads it once per second?

If I were to use the DS1307 square wave output, I’d probably use an interrupt. But that wouldn’t be my first choice unless I had a specific reason for doing it that way. Else I’d use code like above. To implement the square wave, the square wave enable bit (SQWE) in the DS1307 control register has to be set, and the rate select bits (RS1, RS0) have to be set for 1Hz. Basically the control register (address 0x07) just has to be set to 0x10. The SQW/OUT pin is open drain so a pullup resistor is needed; the AVR’s internal pullup could be used.

The other approach is to poll the pin (rapidly) and watch for the transition, say from HIGH to LOW. So the pin is actually read much more often than once a second, but it is the transition that is used as a trigger, and that only happens once per second.

I find it easier to just watch for the time to change as in the code above. I usually need to know the time for other reasons anyway, so it just seems to make sense. No interrupts, and a pin does not have to be dedicated to it.

Thanks for the answers, they are really helpful. I wonder if I can just use ordinary variables for counters instead of the alarm functions which I think I have to download a seperate library for? If I want to log e.g how many hours a day the fan was on.

syphex:
Thanks for the answers, they are really helpful.

Great! You're welcome!

I wonder if I can just use ordinary variables for counters instead of the alarm functions which I think I have to download a seperate library for? If I want to log e.g how many hours a day the fan was on.

I don't know why not, in fact that might be preferable, it doesn't really seem to me like a situation where an alarm is appropriate. I don't hesitate to use libraries though, there's a lot of good stuff out there.

There are various approaches to this. For one, simply log the time whenever the fan is turned on or off. Of course this doesn't tell us how many hours a day the fan was on, but it could be calculated from the logged data. Another approach would use a variable to track the time (e.g. number of seconds) the fan was on. Then once a day, log the value of that variable then zero it. An alarm could be used to do the logging in that case, since it would happen at a particular time each day.

syphex:
I can just use ordinary variables for counters instead of the alarm functions which I think I have to download a seperate library for? If I want to log e.g how many hours a day the fan was on.

Alarm functions are used to control things. It seems to me that you just want to know what a fan is doing, and that means simply logging the data. If you really need alarm functions, I have just started using the library pair, Time and TimeAlarms. They seem excellent.

The arduino activates via relays either a fan or a heater depending of course on the temperature, so the amount of hours a fan or heater is on per day is variable, and will not occur at set times during the day (unless you mean the natural heating and cooling of the earth's surface).

This is the RTC module I ended up getting, Im not sure how it works exactly because there is a p1 and a p2. Are they independant of each other in terms of power and grounding etc? i.e will the square wave pin interfere with the arduino if its connected via the p2 side?

Uploaded with ImageShack.us

I wonder if I can place it inside my arduino like this? Or will it cause interference issues as mentioned above? Also I believe the battery terminal is for an optional battery and isn't required to use the battery at the back of the module? What is it for?

Uploaded with ImageShack.us

Im running out of space on my breadboard :blush: I could rewire it and make some space but it would be neat if I could just sit it in there because its nice and compact plus I can just wire it to the analog pin from there. Maybe if its sitting on top of some header pins?

syphex:
This is the RTC module I ended up getting, Im not sure how it works exactly because there is a p1 and a p2. Are they independant of each other in terms of power and grounding etc? i.e will the square wave pin interfere with the arduino if its connected via the p2 side?

These things should be described on the manufacturer's web site, data sheet, etc.

syphex:
The arduino activates via relays either a fan or a heater depending of course on the temperature, so the amount of hours a fan or heater is on per day is variable, and will not occur at set times during the day

So it appears you are simply timestamping those actions. This is quite straightforward, where you read the clock on an event and record it. No special alarm system warranted.

This is the RTC module I ended up getting, Im not sure how it works exactly because there is a p1 and a p2. Are they independant of each other in terms of power and grounding etc? i.e will the square wave pin interfere with the arduino if its connected via the p2 side?

All you need is four connections Vcc, gnd, sda, scl. The points with the same name are cross-connected so either will do and you can put the module on a proto shield and use it as a sort of junction box.

Also I believe the battery terminal is for an optional battery and isn’t required to use the battery at the back of the module? What is it for?

The battery terminal makes the on-board battery voltage available. I guess you could use it for an off-board battery but I won’t ask why you would do this and not use the on-board battery, which is free and I understand is good for ten years.

it would be neat if I could just sit it in there because its nice and compact plus I can just wire it to the analog pin from there. Maybe if its sitting on top of some header pins?

The obvious thing to do is build it onto a proto shield.

I think your project could benefit from Proportional Integral Derivitave (PID) type control. There are PID libraries available. Take a look at this implementation:

The Uno millis() function can be made to be accurate to within 1ppm simply by multiplying by a constant. There is another thread on here describing exactly how to do it.

I got the DS1307 module wired up and am experiencing some of the problems I anticipated. At the moment I am trying to set the RTC time to the system time and getting the RTC to save the time when the arduino is unplugged.

After downloading the time library I tried the TimeRTCSet example and it seemed to set it to the system time albeit a number of seconds behind. Now I upload the TimeRTC example and it also outputs the correct time to Serial. But when I unplug the Arduino to see if it will use the battery on the RTC module, the date and time have changed when I again run TimeRTC. E.g. it says the time is 17:18:42 2 11 2035, when the time is 19:23:00 7 21 2013.

What is the DS pin used for? And am I supposed to do something with the BAT terminal? Though I understand that is for an optional external battery and should keep the time with the included battery somehow?

You have probably got a timeset routine with a fixed example messing things up.

I submit you are better off having the clock set by a separate programme.

Like this

//Arduino 1.0+ Only
//Arduino 1.0+ Only
// PRE-set the time in the void then USE THE RESET BUTTON to set it!

#include "Wire.h"
#define DS1307_ADDRESS 0x68
byte zero = 0x00; //workaround for issue #527

void setup(){
  Wire.begin();
  Serial.begin(9600);
      
  setDateTime(); //MUST CONFIGURE IN FUNCTION
printDate();
Serial.println("loopstart");
}

void loop(){

  printDate();
  
  delay(1000);
}
// THIS IS WHERE YOU ENTER THE TIME
void setDateTime(){

  byte second =      30; //0-59
  byte minute =      24; //0-59
  byte hour =        0; //0-23
  byte weekDay =     5; //1-7
  byte monthDay =    19; //1-31
  byte month =       4; //1-12
  byte year  =       13; //0-99

  Wire.beginTransmission(DS1307_ADDRESS);
  Wire.write(zero); //stop Oscillator

  Wire.write(decToBcd(second));
  Wire.write(decToBcd(minute));
  Wire.write(decToBcd(hour));
  Wire.write(decToBcd(weekDay));
  Wire.write(decToBcd(monthDay));
  Wire.write(decToBcd(month));
  Wire.write(decToBcd(year));

  Wire.write(zero); //start 

  Wire.endTransmission();

}

byte decToBcd(byte val){
// Convert normal decimal numbers to binary coded decimal
  return ( (val/10*16) + (val%10) );
}

byte bcdToDec(byte val)  {
// Convert binary coded decimal to normal decimal numbers
  return ( (val/16*10) + (val%16) );
}

void printDate(){

  // Reset the register pointer
  Wire.beginTransmission(DS1307_ADDRESS);
  Wire.write(zero);
  Wire.endTransmission();

  Wire.requestFrom(DS1307_ADDRESS, 7);

  int second = bcdToDec(Wire.read());
  int minute = bcdToDec(Wire.read());
  int hour = bcdToDec(Wire.read() & 0b111111); //24 hour time
  int weekDay = bcdToDec(Wire.read()); //0-6 -> sunday - Saturday
  int monthDay = bcdToDec(Wire.read());
  int month = bcdToDec(Wire.read());
  int year = bcdToDec(Wire.read());

  //print the date EG   3/1/11 23:59:59
  Serial.print(month);
  Serial.print("/");
  Serial.print(monthDay);
  Serial.print("/");
  Serial.print(year);
  Serial.print("     ");
  Serial.print(hour);
  Serial.print(":");
  Serial.print(minute);
  Serial.print(":");
  Serial.println(second);
}

The battery keeps time when power is off and should be good for ten years.

You only need four connections; +5v, gnd, sda, scl.

syphex:
the date and time have changed when I again run TimeRTC. E.g. it says the time is 17:18:42 2 11 2035, when the time is 19:23:00 7 21 2013.

What is the DS pin used for? And am I supposed to do something with the BAT terminal? Though I understand that is for an optional external battery and should keep the time with the included battery somehow?