Time and RTC libs - question on date and time integrity

Hi, while playing with Time and RTC libraries (building a simple logger) I came accross following Q:
Is there any mechanism built into the libs which could avoid the situation the Date and Time may desintegrate while reading them around 00:00:00?
I mean the situation where I may read:

a) time_t t=now();
or
b) the sync provider (ie. the setSyncProvider(rtc.get_tm); ) reads the RTC and writes into time_t

while the DATE read is ie. 31/07/2012 and TIME jumps from 23:59:59 to 00:00:00 just before the moment of reading the time related structure (for both internal time_t and external RTC). Then the date&time set will be 31/7/2012 00:00:00, thus a day difference..
p.

All registers in the RTC must be read in the same I2C transaction. I know RTClib GitHub - adafruit/RTClib: A fork of Jeelab's fantastic RTC library reads the DS1307 correctly.

I have written my own libraries for various RTC chips and all Maxim DS I2C chips have this feature:

When reading or writing the time and date registers, secondary (user) buffers are used to prevent errors when the internal registers update. When reading the time and date registers, the user buffers are synchronized to the internal registers on any I2C START. The time information is read from these secondary registers while the clock continues to run. This eliminates the need to re-read the registers in case the internal registers update during a read.

If your library has this bug, just read the time twice until the two values are equal. Better yet find a new library.

pito,

I also wrote my own replacement for Wire for I2C master access to devices so I have better control over the I2C protocol.

I assume Wire does the read correctly as a single transaction but it is very complex with interrupts and both slave and master support.

The Rtc_Pcf8563 library I am using reads Date and Time in two separate transactions. NXP recommends to read in one block to maintain integrity. So it needs to be modified…
http://arduino.cc/playground/Main/RTC-PCF8563
p.

Many RTC chips protect against this in the hardware. For example, from the DS1307 datasheet:

When reading or writing the time and date registers, secondary (user) buffers are used to prevent errors when the
internal registers update. When reading the time and date registers, the user buffers are synchronized to the
internal registers on any I2C START. The time information is read from these secondary registers while the clock
continues to run. This eliminates the need to re-read the registers in case the internal registers update during a
read. The divider chain is reset whenever the seconds register is written. Write transfers occur on the I2C
acknowledge from the DS1307. Once the divider chain is reset, to avoid rollover issues, the remaining time and
date registers must be written within one second.

pito:
The Rtc_Pcf8563 library I am using reads Date and Time in two separate transactions.

For the DS3231, I read it all in one go and process from there. This is what I would always recommend. Perhaps there is a way to combine the reads into one transaction even if the results are later parsed in two different subroutines?

RTClib here https://github.com/jcw/rtclib read all PCF8563 registers correctly:

DateTime RTC_PCF8563::now() {
    Wire.beginTransmission(PCF8563_ADDRESS);
    Wire.write(PCF8563_SEC_ADDR);
    Wire.endTransmission();

    Wire.requestFrom(PCF8563_ADDRESS, 7);
    uint8_t ss = bcd2bin(Wire.read() & 0x7F);
    uint8_t mm = bcd2bin(Wire.read() & 0x7F);
    uint8_t hh = bcd2bin(Wire.read() & 0x3F);
    uint8_t d = bcd2bin(Wire.read() & 0x3F);
    Wire.read();
    uint8_t m = bcd2bin(Wire.read()& 0x1F);
    uint16_t y = bcd2bin(Wire.read()) + 2000;
    
    return DateTime (y, m, d, hh, mm, ss);
}

RTClib also works with DS1307. DS1307 libraries often work with some ot the other Maxim I2C RTC chips.

Maybe we need an “RTC” library (or it shall be a part of Time.cpp) with all that functionality you may find with various rtc_chip’s libs, and the libs related to the chips shall include only low level routines for reading/writing datetime and alarm…
For example ds1307 and pcf libs differ in number of various datetime conversion functions, even the chips are basically the same except the registers order…
p.

Enclosed pls find the modified library for PCF8563 ( see http://arduino.cc/playground/Main/RTC-PCF8563 ).
Added getDateAndTime() for oneshot regs read, and get_tm() for usage with ie. setSyncProvider(rtc.get_tm)…
p.

Rtc_Pcf8563.h (4.31 KB)

Rtc_Pcf8563.cpp (14.5 KB)

I wrote several RTC GPS date time libraries.

The most complete library uses UTC for RTC chips and GPS. It handles time zones and daylight/summer time. It can format date time in ISO formats or other custom formats. It has all popular POSIX conversions between epoch time and local/UTC. It has a database and algorithms for common DST rules.

This library is overkill for most users so I never posted it.

Some uses need a very lean library for data logging that just gives yyyy-mm-dd hh:mm:ss from the RTC registers. Other users need more features.

My guess that existing libraries satisfy most user's needs if you get rid of bugs.

Date time libraries can be a bottomless pit. Just look a libraries like Boost Date Time.

This is more of a problem of knowing what users need than writing code. Once you know the requirements code is easy.

One positive thing about the Time.cpp library is it uses "time_t" what is the unix_time. So the rtc chip library needs to convert the rtc date and time to time_t, all the other functions are various formatting exercises around time_t only..
So the rtc_chip_lib shall include ~4 functions only: getDateAndTime(), setDateAndTime() and the same with Alarm..

The Time library does not produce Unix/POSIX time. The standard for the POSIX epoch is 1970-01-01 00:00:00 UTC.

Not the number of seconds from some random local time. POSIX time is the same in every time zone at a given instant.

This is what I get (UTC):
07/31/2012 22:29:58 1343773798
p.

So what? Time can not convert between POSIX time and local time since it doesn't know about time zones. It is not a POSIX time library.

You must be two hours east of Zulu time. Well one hour plus summer time.

fat16lib:
I wrote several RTC GPS date time libraries.

The most complete library uses UTC for RTC chips and GPS. It handles time zones and daylight/summer time. It can format date time in ISO formats or other custom formats. It has all popular POSIX conversions between epoch time and local/UTC. It has a database and algorithms for common DST rules.

This library is overkill for most users so I never posted it.

....

This is more of a problem of knowing what users need than writing code. Once you know the requirements code is easy.

Several months ago, I bought a do-it-yourself alarm clock called the Alpha Clock Five. I bought it with the intention of modifying it to show the date as well as the time, and also to show seconds. I found it immensely difficult to modify to my liking. Maybe I'm not much of a coder.

What I would have found extremely useful during the process would have been a library like the one you describe. What I would like to know, though, is two things:

  • How fast does your library run? This clock uses a multiplexed LED display: would I see constant flickering while the time is being recalculated?
  • I made a display mode which shows the time in increments of 1/8 minute. This is 7.5 seconds per increment. Can your library handle the half second?

Please post your library so I can have a look at it.

Some functions I have written are here: Arduino Forum

A couple of observations...

  1. Use the 1Hz signal coming out of the RTC to drive one of the two interrupt pins on the 328P and hence increment the seconds. Thus, you do not have to query the clock at all via I2C to get great second-by-second performance. Sure, you have to keep track of seconds and minutes, and hours, but that's pretty simple.

  2. If you want to be extra careful (guard against bad code or other mishaps resulting in a lost interrupt), have the code query the clock once a day in the middle of the night, etc. Then your seconds, minutes, and hours, will once again be in perfect harmony.

  3. Never worry about flickering again - the code overhead for the clock is hence minimized and the Arduino can spend its time keeping the display happy.

Constantin:
A couple of observations...

  1. Use the 1Hz signal coming out of the RTC to drive one of the two interrupt pins on the 328P and hence increment the seconds. Thus, you do not have to query the clock at all via I2C to get great second-by-second performance. Sure, you have to keep track of seconds and minutes, and hours, but that's pretty simple.

  2. If you want to be extra careful (guard against bad code or other mishaps resulting in a lost interrupt), have the code query the clock once a day in the middle of the night, etc. Then your seconds, minutes, and hours, will once again be in perfect harmony.

  3. Never worry about flickering again - the code overhead for the clock is hence minimized and the Arduino can spend its time keeping the display happy.

I'm sorry if I didn't make myself clear. What I aim to do is to have the clock show civil time, with all its irregularities, and to be as independent from outside sources of time-related information as possible. I am using a Chronodot, but do not count this as an "outside source"; the outside sources to which I refer are mains frequency and NTP / radio / GPS signals. Ideally, I set the clock once in the entire duration of its existence, though in practice, I will almost certainly need to make occasional adjustments of a few seconds.

I would prefer to be able to tell the clock, "leap second coming at end of month XXX", and have it correctly insert that at the right time, as well.

odometer:
I'm sorry if I didn't make myself clear. What I aim to do is to have the clock show civil time, with all its irregularities, and to be as independent from outside sources of time-related information as possible. I am using a Chronodot, but do not count this as an "outside source"; the outside sources to which I refer are mains frequency and NTP / radio / GPS signals. Ideally, I set the clock once in the entire duration of its existence, though in practice, I will almost certainly need to make occasional adjustments of a few seconds.

I would prefer to be able to tell the clock, "leap second coming at end of month XXX", and have it correctly insert that at the right time, as well.

Pardon me for asking, but what part of my recommendation runs counter to your stated desires for this project? You wanted to avoid flickering displays, I addressed that issue by minimizing the CPU overhead associated with querying the Chronodot to once a day. That configuration also allows you maximum CPU capacity for other features, such as showing seconds in 1/2 increments.

If you are not familiar with the SQW output that the Chronodot features, now may be a good time to re-read the DS3231 datasheet and Atmel 328P interrupts. I have used the SQW output for my projects, it makes timekeeping much easier without a lot of overhead. The only downside as I see it are the (IIRC) 51-odd instruction cycles that it takes to drop into a interrupt routine. That said, 51 out 16 Million cycles per second is a pretty small percentage allowing your MCU to do other things 99.9997% of the time.

With careful monitoring and the adjustment of the aging trim (see datasheet) you may be able to minimize clock drift.

As far as the insertion of leap seconds go, here is a quote from Wikipedia (for what that's worth):

Because the Earth's rotation speed varies in response to climatic and geological events, UTC leap seconds are irregularly spaced and unpredictable. Insertion of each UTC leap second is usually decided about six months in advance by the International Earth Rotation and Reference Systems Service (IERS), when needed to ensure that the difference between the UTC and UT1 readings will never exceed 0.9 second. Between their adoption in 1972 and June 2012, 25 leap seconds have been scheduled, all positive.

(emphasis mine).

So in forty years, 25 seconds have been added at irregular intervals. My suggestion would be to worry about clock drift instead, the DS3231 is not nearly that accurate. Ideally, it'll be around 2PPM without accounting for aging, reflow soldering, etc. That's 63 seconds per year or ~5 seconds per month, assuming a non-random distribution of the error.

If I had too much time on my hands, it would make for an interesting experiment to see how accurate the DS3231 can be made through careful adjustment of the aging trim. For example, by connecting it to a radio time stamp circuit and comparing the 'should' vs. 'actual' seconds elapsed over a two month period and auto-adjusting the trim accordingly.

Let me spell it out for you, Constantin: Daylight Saving Time.

What I am thinking is, have the "external" RTC keep UTC, and convert for display.

Hi odometer, thanks for the clarification. Daylight Savings Time is a bit more precise description than “civil” time. Implementing it should be easy.

Implementing auto-leap-second insertion as you stated earlier should be more difficult for the reasons I outlined above.

Looking over the schematic, I see that the Evil Mad Science alpha clock does not seem to have an connection from the SQW output from the Chronodot to one of the 644 hardware interrupts. Oh well, there goes that great idea.

IMO, no library is needed to access all the data in the Chronodot. Implementing the I2Ccode is pretty simple, gives you a great appreciation for bit shifting and other operations to assemble the various pieces of data.

Best of luck with your project.