Time and RTC libs - question on date and time integrity

The Rtc_Pcf8563 library I am using reads Date and Time in two separate transactions. NXP recommends to read in one block to maintain integrity. So it needs to be modified..
http://arduino.cc/playground/Main/RTC-PCF8563
p.

Many RTC chips protect against this in the hardware. For example, from the DS1307 datasheet:

When reading or writing the time and date registers, secondary (user) buffers are used to prevent errors when the
internal registers update. When reading the time and date registers, the user buffers are synchronized to the
internal registers on any I2C START. The time information is read from these secondary registers while the clock
continues to run. This eliminates the need to re-read the registers in case the internal registers update during a
read. The divider chain is reset whenever the seconds register is written. Write transfers occur on the I2C
acknowledge from the DS1307. Once the divider chain is reset, to avoid rollover issues, the remaining time and
date registers must be written within one second.

pito:
The Rtc_Pcf8563 library I am using reads Date and Time in two separate transactions.

For the DS3231, I read it all in one go and process from there. This is what I would always recommend. Perhaps there is a way to combine the reads into one transaction even if the results are later parsed in two different subroutines?

RTClib here https://github.com/jcw/rtclib read all PCF8563 registers correctly:

DateTime RTC_PCF8563::now() {
    Wire.beginTransmission(PCF8563_ADDRESS);
    Wire.write(PCF8563_SEC_ADDR);
    Wire.endTransmission();

    Wire.requestFrom(PCF8563_ADDRESS, 7);
    uint8_t ss = bcd2bin(Wire.read() & 0x7F);
    uint8_t mm = bcd2bin(Wire.read() & 0x7F);
    uint8_t hh = bcd2bin(Wire.read() & 0x3F);
    uint8_t d = bcd2bin(Wire.read() & 0x3F);
    Wire.read();
    uint8_t m = bcd2bin(Wire.read()& 0x1F);
    uint16_t y = bcd2bin(Wire.read()) + 2000;
    
    return DateTime (y, m, d, hh, mm, ss);
}

RTClib also works with DS1307. DS1307 libraries often work with some ot the other Maxim I2C RTC chips.

Maybe we need an "RTC" library (or it shall be a part of Time.cpp) with all that functionality you may find with various rtc_chip's libs, and the libs related to the chips shall include only low level routines for reading/writing datetime and alarm..
For example ds1307 and pcf libs differ in number of various datetime conversion functions, even the chips are basically the same except the registers order..
p.

Enclosed pls find the modified library for PCF8563 ( see Arduino Playground - Rtc_Pcf8563 RTC Library ).
Added getDateAndTime() for oneshot regs read, and get_tm() for usage with ie. setSyncProvider(rtc.get_tm)..
p.

Rtc_Pcf8563.h (4.31 KB)

Rtc_Pcf8563.cpp (14.5 KB)

I wrote several RTC GPS date time libraries.

The most complete library uses UTC for RTC chips and GPS. It handles time zones and daylight/summer time. It can format date time in ISO formats or other custom formats. It has all popular POSIX conversions between epoch time and local/UTC. It has a database and algorithms for common DST rules.

This library is overkill for most users so I never posted it.

Some uses need a very lean library for data logging that just gives yyyy-mm-dd hh:mm:ss from the RTC registers. Other users need more features.

My guess that existing libraries satisfy most user's needs if you get rid of bugs.

Date time libraries can be a bottomless pit. Just look a libraries like Boost Date Time.

This is more of a problem of knowing what users need than writing code. Once you know the requirements code is easy.

One positive thing about the Time.cpp library is it uses "time_t" what is the unix_time. So the rtc chip library needs to convert the rtc date and time to time_t, all the other functions are various formatting exercises around time_t only..
So the rtc_chip_lib shall include ~4 functions only: getDateAndTime(), setDateAndTime() and the same with Alarm..

The Time library does not produce Unix/POSIX time. The standard for the POSIX epoch is 1970-01-01 00:00:00 UTC.

Not the number of seconds from some random local time. POSIX time is the same in every time zone at a given instant.

This is what I get (UTC):
07/31/2012 22:29:58 1343773798
p.

So what? Time can not convert between POSIX time and local time since it doesn't know about time zones. It is not a POSIX time library.

You must be two hours east of Zulu time. Well one hour plus summer time.

fat16lib:
I wrote several RTC GPS date time libraries.

The most complete library uses UTC for RTC chips and GPS. It handles time zones and daylight/summer time. It can format date time in ISO formats or other custom formats. It has all popular POSIX conversions between epoch time and local/UTC. It has a database and algorithms for common DST rules.

This library is overkill for most users so I never posted it.

....

This is more of a problem of knowing what users need than writing code. Once you know the requirements code is easy.

Several months ago, I bought a do-it-yourself alarm clock called the Alpha Clock Five. I bought it with the intention of modifying it to show the date as well as the time, and also to show seconds. I found it immensely difficult to modify to my liking. Maybe I'm not much of a coder.

What I would have found extremely useful during the process would have been a library like the one you describe. What I would like to know, though, is two things:

  • How fast does your library run? This clock uses a multiplexed LED display: would I see constant flickering while the time is being recalculated?
  • I made a display mode which shows the time in increments of 1/8 minute. This is 7.5 seconds per increment. Can your library handle the half second?

Please post your library so I can have a look at it.

Some functions I have written are here: http://arduino.cc/forum/index.php/topic,101401.0.html

A couple of observations...

  1. Use the 1Hz signal coming out of the RTC to drive one of the two interrupt pins on the 328P and hence increment the seconds. Thus, you do not have to query the clock at all via I2C to get great second-by-second performance. Sure, you have to keep track of seconds and minutes, and hours, but that's pretty simple.

  2. If you want to be extra careful (guard against bad code or other mishaps resulting in a lost interrupt), have the code query the clock once a day in the middle of the night, etc. Then your seconds, minutes, and hours, will once again be in perfect harmony.

  3. Never worry about flickering again - the code overhead for the clock is hence minimized and the Arduino can spend its time keeping the display happy.

Constantin:
A couple of observations...

  1. Use the 1Hz signal coming out of the RTC to drive one of the two interrupt pins on the 328P and hence increment the seconds. Thus, you do not have to query the clock at all via I2C to get great second-by-second performance. Sure, you have to keep track of seconds and minutes, and hours, but that's pretty simple.

  2. If you want to be extra careful (guard against bad code or other mishaps resulting in a lost interrupt), have the code query the clock once a day in the middle of the night, etc. Then your seconds, minutes, and hours, will once again be in perfect harmony.

  3. Never worry about flickering again - the code overhead for the clock is hence minimized and the Arduino can spend its time keeping the display happy.

I'm sorry if I didn't make myself clear. What I aim to do is to have the clock show civil time, with all its irregularities, and to be as independent from outside sources of time-related information as possible. I am using a Chronodot, but do not count this as an "outside source"; the outside sources to which I refer are mains frequency and NTP / radio / GPS signals. Ideally, I set the clock once in the entire duration of its existence, though in practice, I will almost certainly need to make occasional adjustments of a few seconds.

I would prefer to be able to tell the clock, "leap second coming at end of month XXX", and have it correctly insert that at the right time, as well.

odometer:
I'm sorry if I didn't make myself clear. What I aim to do is to have the clock show civil time, with all its irregularities, and to be as independent from outside sources of time-related information as possible. I am using a Chronodot, but do not count this as an "outside source"; the outside sources to which I refer are mains frequency and NTP / radio / GPS signals. Ideally, I set the clock once in the entire duration of its existence, though in practice, I will almost certainly need to make occasional adjustments of a few seconds.

I would prefer to be able to tell the clock, "leap second coming at end of month XXX", and have it correctly insert that at the right time, as well.

Pardon me for asking, but what part of my recommendation runs counter to your stated desires for this project? You wanted to avoid flickering displays, I addressed that issue by minimizing the CPU overhead associated with querying the Chronodot to once a day. That configuration also allows you maximum CPU capacity for other features, such as showing seconds in 1/2 increments.

If you are not familiar with the SQW output that the Chronodot features, now may be a good time to re-read the DS3231 datasheet and Atmel 328P interrupts. I have used the SQW output for my projects, it makes timekeeping much easier without a lot of overhead. The only downside as I see it are the (IIRC) 51-odd instruction cycles that it takes to drop into a interrupt routine. That said, 51 out 16 Million cycles per second is a pretty small percentage allowing your MCU to do other things 99.9997% of the time.

With careful monitoring and the adjustment of the aging trim (see datasheet) you may be able to minimize clock drift.

As far as the insertion of leap seconds go, here is a quote from Wikipedia (for what that's worth):

Because the Earth's rotation speed varies in response to climatic and geological events, UTC leap seconds are irregularly spaced and unpredictable. Insertion of each UTC leap second is usually decided about six months in advance by the International Earth Rotation and Reference Systems Service (IERS), when needed to ensure that the difference between the UTC and UT1 readings will never exceed 0.9 second. Between their adoption in 1972 and June 2012, 25 leap seconds have been scheduled, all positive.

(emphasis mine).

So in forty years, 25 seconds have been added at irregular intervals. My suggestion would be to worry about clock drift instead, the DS3231 is not nearly that accurate. Ideally, it'll be around 2PPM without accounting for aging, reflow soldering, etc. That's 63 seconds per year or ~5 seconds per month, assuming a non-random distribution of the error.

If I had too much time on my hands, it would make for an interesting experiment to see how accurate the DS3231 can be made through careful adjustment of the aging trim. For example, by connecting it to a radio time stamp circuit and comparing the 'should' vs. 'actual' seconds elapsed over a two month period and auto-adjusting the trim accordingly.

Let me spell it out for you, Constantin: Daylight Saving Time.

What I am thinking is, have the "external" RTC keep UTC, and convert for display.

Hi odometer, thanks for the clarification. Daylight Savings Time is a bit more precise description than "civil" time. Implementing it should be easy.

Implementing auto-leap-second insertion as you stated earlier should be more difficult for the reasons I outlined above.

Looking over the schematic, I see that the Evil Mad Science alpha clock does not seem to have an connection from the SQW output from the Chronodot to one of the 644 hardware interrupts. Oh well, there goes that great idea.

IMO, no library is needed to access all the data in the Chronodot. Implementing the I2Ccode is pretty simple, gives you a great appreciation for bit shifting and other operations to assemble the various pieces of data.

Best of luck with your project.

Constantin:
Hi odometer, thanks for the clarification. Daylight Savings Time is a bit more precise description than "civil" time. Implementing it should be easy.

To do DST right, I need to be able to do date stuff right.
I find date stuff tricky to code and trickier to test.
Maybe it's because I'm really not used to working in this kind of environment (C++, microcontroller). I've done plenty of date / time related stuff on my own, and I believe that I am at least aware of the major "gotchas" by now. But I really am not perfect.

I think I might do this:
Handle the "messy stuff" only once every, say, 10 minutes. So the routine would go something like this.
Let's say I power the clock on. The RTC says that it is 12:34:56 (this is UTC) on such-and-such a date in the part of the year with DST. The program assumes it is the middle of the second (so as not to be more than 1/2 second out), and performs the conversion to local time for display: the result is 08:34:56.5. So I see "8:34:56" for half a second, then "8:34:57" for one full second, then "8:34:58" for one full second,... During this process, the RTC is ignored, and the half-seconds, seconds, and minutes are counted "stupidly". This continues until any XX:X9:59.5, at which time the program reads the RTC seconds once every 10 or 20 milliseconds. If it sees "59", it stands still. If it sees "00", it moves on. So, to continue our example, when our program reaches 08:39:59.5, it reads the RTC seconds, and one of three things happens:
a) The RTC seconds are "59" (or possibly "58"). Our program stays put at 08:39:59.5.
b) The RTC seconds are "00". Our program performs a full recalculation of the local time. It is now 08:40:00.0. (I am not sure whether this case involved reading more than just the seconds from the RTC, but it would probably be prudent.)
c) The RTC seconds are something other than "58", "59", or "00". Our program screams bloody murder.
The reason for the "full recalculation of the local time" in (b) is because sometimes there will be a discontinuity, such as when Daylight Saving Time begins or ends. Also, this recalculation will give us the local date, which is not always the same as the RTC date. Remember, I have my clock programmed (more like jury-rigged) to show the date as well as the time.

While I'm at it, I will post my (horrible) modification of the Alpha Clock Five firmware:

(Edit: There is a big bug in the code for the alarm. Depending on conditions too detailed to list, it might or might not work. Do not rely on the alarm!)

firmware_17a_adj.pde (53.2 KB)