Result of strftime() is 30 years off?

Hi,

I need to convert an arbitrary UnixTime code to a string in a prescribed format.
This is the simple sketch I have, using strftime():

#include <time.h>

void setup(void) {
  Serial.begin(SERIAL_SPEED);
  while (!Serial) {
    ;
  }

  time_t t = 1591050067;
  struct tm *tm = localtime(&t);
  char dts[22];
  strftime(dts, sizeof(dts), "%Y-%m-%d - %X", tm);
  Serial.println(dts);

}

The strange thing is that the result is:
“2050-06-01 - 22:21:07” but this should be “2020-06-01 - 22:21:07”.
So it is 30 years off?
The exact same method works fine in CodeBlocks.

Does anybody have an idea what is going on here?

Which board are you compiling for?

Thanks.

I tried it both with an Arduino Uno (original) and an Arduino Nano (clone with old bootloader).

The problem seems to be in the conversion from 'time_t' to 'struct tm' in localtime(). The resulting 'struct tm' has the 'tm_year' field ("number of years since 1900") set to 150 which indicates the year 2050.

Thanks John.
Unfortunately I am not experienced enough to fully understand your answer.
Does this mean that it is a bug in the library?
Do you know of any way to go around this?

From <time.h>: Time:

Though not specified in the standard, it is often expected that time_t is a signed integer representing an offset in seconds from Midnight Jan 1 1970... i.e. 'Unix time'. This implementation uses an unsigned 32 bit integer offset from Midnight Jan 1 2000. The use of this 'epoch' helps to simplify the conversion functions, while the 32 bit value allows time to be properly represented until Tue Feb 7 06:28:15 2136 UTC. The macros UNIX_OFFSET and NTP_OFFSET are defined to assist in converting to and from Unix and NTP time stamps.

(Emphasis mine.)

A good overview:
http://www.catb.org/esr/time-programming/

Use the TimeLib library and the breakTime() function. The year element will be years since 1970:

#include <TimeLib.h>

void setup(void) {
  tmElements_t convertedTime;
  Serial.begin(115200);
  delay(1000);

  time_t unixTime = 1591050067;
  breakTime(unixTime, convertedTime);
  Serial.print(convertedTime.Year+1970);
  Serial.print("-");
  Serial.print(convertedTime.Month);
  Serial.print("-");
  Serial.print(convertedTime.Day);
  Serial.print("-");
  Serial.print(convertedTime.Hour);
  Serial.print("-");
  Serial.print(convertedTime.Minute);
  Serial.print("-");
  Serial.println(convertedTime.Second);
}

void loop() {
}

gfvalvo:
Use the TimeLib library and the breakTime() function. The year element will be years since 1970:

#include <TimeLib.h>

void setup(void) {
 tmElements_t convertedTime;
 Serial.begin(115200);
 delay(1000);

time_t unixTime = 1591050067;
 breakTime(unixTime, convertedTime);
 Serial.print(convertedTime.Year+1970);
 Serial.print("-");
 Serial.print(convertedTime.Month);
 Serial.print("-");
 Serial.print(convertedTime.Day);
 Serial.print("-");
 Serial.print(convertedTime.Hour);
 Serial.print("-");
 Serial.print(convertedTime.Minute);
 Serial.print("-");
 Serial.println(convertedTime.Second);
}

void loop() {
}

Looks like TimeLib (at least the functionality you used in your example) does basically the same thing that <time.h> does but in a non-standard way.

No need to convert OP's code to use a different (and non-standard) library. Just use <time.h> and subtract UNIX_OFFSET from the Unix timestamp to get a valid AVR time.h timestamp:

time_t t = 1591050067 - UNIX_OFFSET;

christop:
Looks like TimeLib (at least the functionality you used in your example) does basically the same thing that <time.h> does but in a non-standard way.

No need to convert OP's code to use a different (and non-standard) library.

TimeLib is the standard time library of the Arduino Ecosystem.

If, after all that, the time is then just a few hours out, then look at the timezone library for local time (including daylight saving time) correction.

Thanks to you all guys, this really helped me! :slight_smile:

I think I will use the UNIX_OFFSET with the standard time.h library.
Just one more question to be sure: does this UNIX_OFFSET keep on working correctly through all coming leap years?

Update:
Never mind: I already found out that indeed this is the case. :slight_smile:

IMO, the avr implementation is really broken by changing the epoch.
While the time_t epoch is not defined by the C standard it has been consistent through now 50 years of unix history and is defined by POSIX, which is a standard.

--- bill

bperrybap:
IMO, the avr implementation is really broken by changing the epoch.
While the time_t epoch is not defined by the C standard it has been consistent through now 50 years of unix history and is defined by POSIX, which is a standard.

--- bill

I think the rationale behind using an epoch of January 1, 2000 rather than POSIX's epoch is that

  • AVR isn't POSIX, so it's not bound by POSIX rules,
  • it allows small microcontroller applications to continue tracking time well past 2038 (the 32-bit Unix "end of time") without switching to a larger size for the time_t type (and time_t is unsigned on AVR because most microcontroller applications generally don't have to handle times from last century), and
  • I think it also simplifies some of the calculations.

The epoch (and unsigned time_t) used by avr-gcc effectively moves the supported time range forward by nearly 100 years (2000–2136 instead of 1902–2038). The biggest downside to this is that an application must add/subtract a fixed offset to convert time to/from Unix time.

christop:
I think the rationale behind using an epoch of January 1, 2000 rather than POSIX's epoch is that

  • AVR isn't POSIX, so it's not bound by POSIX rules,
  • it allows small microcontroller applications to continue tracking time well past 2038 (the 32-bit Unix "end of time") without switching to a larger size for the time_t type (and time_t is unsigned on AVR because most microcontroller applications generally don't have to handle times from last century), and
  • I think it also simplifies some of the calculations.

The epoch (and unsigned time_t) used by avr-gcc effectively moves the supported time range forward by nearly 100 years (2000–2136 instead of 1902–2038). The biggest downside to this is that an application must add/subtract a fixed offset to convert time to/from Unix time.

Possibly, but, IMO, they are not good rational particularly since what they have done in #3 breaks the time functions API and makes it not portable with every other implementation. IMO, breaking a well known API is a very bad thing.
Moving the epoch wasn't what got them beyond 2038. That was done by changing the time_t to a unsigned value which does not affect the epoch or timestamp API compatibility.
They changed the epoch for an apparent "simplification" of leap year calculations in the libC code.

  1. POSIX compliance
    Sure AVR libC isn't POSIX, but look at all the other functions in AVR libC that conform to the traditional unix APIs and expected POSIX API definitions.
    Once you start to deviate from well known APIs that users have used for decades and have come to expect and are still using today, you are creating problems for users.

  2. The 2038 problem
    They did switch to using a 32 bit unsigned integer for time_t. This is a good thing.
    (This what the Arduino Time/TimeLib as well as many other 32 bit implementations have done)
    It maintains the expected API with the 1970 epoch for time_t values and maintains full backward compatibility.
    It shifts the supported time frame to be 1970 to 2106.
    This solves the most critical issue of the code breaking in 2038.
    Losing time tracking for dates prior to 1970 is a non issue for many devices that don't ever need to track events prior to 1970.

  3. Shifting the epoch to simplify the code
    IMO, this was a dumb thing to do.
    IMO, it wasn't needed and there are other ways to handle their code "simplifications" than shifting the actual epoch.
    The only place where "simplified" code would come into play is when the time_t is broken down into its components.
    There is a comment in the AVR libC code mentioning that because of their epoch, it simplifies the leap year calculation since 2000 (their epoch) is:

at the conjunction of all three 'leap cycles' * 4, 100, and 400 years ( though we can ignore the 400 year cycle in this library).

While it does make the leap year calculation a bit simpler (mostly faster not really simpler or that much smaller), the leap year portion of that (which is where AVR libC code says they have simplified things) is a very small portion of the code.
To see how this is done look at gmmtime_r() in gmtime_r.c in the AVR libC code library.
Then compare that code to the code in the breakTime() function in TimeLib which supports time from 1970 to 2106 using the standard unix time_t epoch of 1970.

Doing what they did in AVR libC may have sped up the break down code a bit, but it came at the expense of API compatibility.

There was no need shift the epoch which breaks API compatibility for a function that is typically not time critical and don't think that gaining the additional 30 years on the backend is that important for dates that are 100 years from now.

Even if they wanted to limit the time frame to dates from year 2000 and beyond, they could have hidden that internally
and used normal time_t values external to the library.
i.e. offset the time_t value just before returning it or just before using it.
or just not supported dates between 1970 and 2000 (to avoid the leap year calculations) and just said that the library only supports dates from year 2000 and will return nonsense if you attempt to use a time_t that is prior to 2000.

The issue with the way they have done it is that it is not compatible with all the other implementations even though the API function names and arguments are the same as the other implementations.
So if you write code for the AVR implementation, and then move it to another processor, the code breaks or doesn't work the same. And if you have a system that uses multiple processors, you can't directly compare timestamps unless you put in library specific conditionals to handle it.
You also have issues with other Arduino libraries that use and expect normal unix epoch time_t values.
While all this can be accounted for, it is a pain.

IMO, what they did (shifting the epoch) is not a useful benefit to the user and while it potentially creates surprise issues, just like what we saw in this thread.

--- bill

Yes, I agree it would have been nice to match the API of some other, pre-existing systems, but it is what it is. The relevant standards that actually define the API used in the AVR world (i.e., C/C++, not POSIX, since that doesn't apply) don't even specify the epoch used by time functions; heck, they don't even specify that time_t is an integral type! I guess one of the lessons is that programmers shouldn't assume something that is not specified.

As far as the communicating timestamps between processors in a multi-processor system, I see that as a communication protocol problem. As part of the system's design, pick a common timestamp format and then convert to/from that format to the application's native format in the application running on each processor; if the native format happens to match the common format, the conversion is a no-op. It's not a much different problem from handling communication between processors with differing endianness (the "standard" way to do that, at least over Internet protocols, is to transmit big end first).

(I've personally found that developing for multiple different platforms teaches you how to write less buggy and more portable software by not making assumptions that just happen to apply to one or more of the platforms but not to others. See word size, endianness, floating-point format (IEEE 754 is not completely universal), null pointer value (it's not zero everywhere), pointer size, size of char (it's 9 bits wide on some platforms and 32 on some other platform and ...), signed value representation (the world is not all two's complement), order of struct bit fields, struct padding, stack growth direction, etc.)

christop, I hear you.
I am very familiar with coding techniques and practices for portability.
I've been writing firmware for 40 years and have writing C code for 37. I've written LOTS of code that had to run on MANY different platforms including some code that was shared across very diverse environments like bare metal, Windows, and unix, including between a host and an intelligent i/o card where there were endian differences.
I've used lots of type definitions, macros, and layering techniques and abstraction layers through the years to ensure code portability across platforms, compiler tool sets, and library implementations.
I even have code that I wrote back in the 80's that I have up and running on AVRs.
So yeah, I'm very familiar with portability techniques, issues and best practices.

All that said, things tend to be much simpler these days vs the 80's in terms of compiler, type and and library API portability with more recent standards.

I'm also very familiar with many implementations of the unix time functions.
I've seen all kinds of crazy and goofy things over the decades.

But you have to admit that changing the epoch from 1970 to 2000 was a pretty silly thing to do, especially given how recent this code was written. (around 2012 and then modified for this change in 2013)
It is not even consistent with the AVR libC library statement about its main goal for design & portability:

In general, it has been the goal to stick as best as possible to established standards while implementing this library.
Commonly, this refers to the C library as described by the ANSI X3.159-1989 and ISO/IEC 9899:1990 ("ANSI-C")
standard, as well as parts of their successor ISO/IEC 9899:1999 ("C99"). Some additions have been inspired by
other standards like IEEE Std 1003.1-1988 ("POSIX.1"), while other extensions are purely AVR-specific (like the
entire program-space string interface).

While the time_t epoch is not in the C standard it is part of POSIX as well as having been used now for 50 years so it is definitely part of well "established standards".

In the end, changing the epoch from its well established value was a silly change that didn't buy much but created an API timestamp compatibility issue that can catch people by surprise as we have seen here in this thread.

I ran across this very same issue a few years ago so I new the issue immediately when I saw the title of the thread.

--- bill

...but as I asked in a previous thread, "what is a programmer to do?". Not only do you have to make some choice among time implementations, but you have to be concerned about 2036, as it is now only 16 years away.

aarg:
...but as I asked in a previous thread, "what is a programmer to do?". Not only do you have to make some choice among time implementations, but you have to be concerned about 2036, as it is now only 16 years away.

The year 2038 wrap around point issue AKA "Y2K38" is a real issue.
There is a lot of stuff that is going to break or go bonkers January 19, 2038.
This is a much bigger issue than Y2K.

For the most part, the 2038 issue (which is only for 32 bit time_t values) can be resolved with using a 32 bit unsigned value that some time libraries (at least for embedded platforms) like the arduino TimeLib has already implemented.
That buys some time up to 2106.
Which is likely to be beyond the life of a device created using this 32 bit time_t limitation or system apps will have been rewritten to use 64 bit time_t values.

But it is an issue right now for many bundled time libraries that come with embedded toolsets like those for Arduino.
Here is a few I've looked at.
AVR core time library uses unsigned 32 bit in so no issues (other than the wrong epoch) until 2136
chipkit32 core time library has Y2K38 issue
esp8266 core time library has Y2K38 issue
esp32 core time library has Y2K38 issue
STM32 core uses 64 bit time_t values so no issue there.

--- bill

Hi,

Thanks again for all the useful responses, and it was also interesting to read the background information about the Arduino history and future problems.

In the last two days I modified my Arduino code, putting the mentioned UnixTime to string conversion in a function, because I have to use this code several times in my current project.
Initially I did use the time.h library with the UNIX_OFFSET option, and after a lot of trial and error I finished with the following three variants:

// Variant 1, using time.h and String
#include <time.h>
#include <locale.h>

// Function to convert a UnixTime value to a formatted string using time.h
String getDateTimeString( long ut ) {
  time_t t = ut - UNIX_OFFSET;
  struct tm * tm = localtime(&t);
  char dts[22];
  strftime(dts, sizeof(dts), "%F - %T", tm);
  return dts;
}

void setup(void) {
  Serial.begin(57600);
  while (!Serial) {
    ;
  }
  long UnixTime = 1591107846;
  String dateTime = getDateTimeString(UnixTime);
  Serial.println(dateTime);
}

void loop() {
  // put your main code here, to run repeatedly:

}
// Output:    2020-06-02 - 14:24:06
// Sketch: 7328 bytes on Arduino Uno
// Global vars: 330 bytes on Arduino Uno
// Variant 2, using time.h and char
#include <time.h>
#include <locale.h>

// Function to convert a UnixTime value to a formatted Char using time.h
char * getDateTimeChar( long ut ) {
  time_t t = ut - UNIX_OFFSET;
  struct tm * tm = localtime(&t);
  static char dtc[22];
  strftime(dtc, sizeof(dtc), "%F - %T", tm);
  return dtc;
}

void setup(void) {
  Serial.begin(57600);
  while (!Serial) {
    ;
  }
  long UnixTime = 1591107846;
  char * dateTime = getDateTimeChar(UnixTime);
  Serial.println(dateTime);
}

void loop() {
  // put your main code here, to run repeatedly:

}
// Output:   2020-06-02 - 14:24:06
// Sketch: 6284 bytes on Arduino Uno
// Global vars: 342 bytes on Arduino Uno
// Variant 3, using time.h and char, and isotime()
#include <time.h>
#include <locale.h>

// Function to convert a UnixTime value to a formatted ISO-Time string using time.h
char * getIsoTimeChar(long ut) {
  time_t t = ut - UNIX_OFFSET;
  struct tm * tm = localtime(&t);
  char * it = isotime(tm);
  return it;
}

void setup(void) {
  Serial.begin(57600);
  while (!Serial) {
    ;
  }
  long UnixTime = 1591107846;
  char * dateTime = getIsoTimeChar(UnixTime);
  Serial.println(dateTime);
}

void loop() {
  // put your main code here, to run repeatedly:

}
// Output:   2020-06-02 14:24:06
// Sketch: 2642 bytes on Arduino Uno
// Global vars: 234 bytes on Arduino Uno

Especially variant 3 is interesting due to the low resources and line count, but a minor disadvantage in my case is that the intermediate dash is missing in the ISO string.

But for using the time.h library I had to include it together with the locale.h as extra includes in my current project. And because I am already running low on resources of my Arduino Mega 2560 with my current project, which already is using the TimeLib.h library for some other applications, I decided to have another go with the TimeLib.h library instead.
And again after a lot of trial and error I finished with the following additional two variants:

// Variant 4, using TimeLib.h and String
#include <TimeLib.h>

// Function to always return Two Digits for Date and Time using String
String twoDigits(String digits) {
  if ( digits.length() < 2 ) {
    digits = "0" + digits;
  }
  return digits;
}

// Function to convert a UnixTime value to a formatted String using TimeLib.h
String getDateTimeString( time_t ut ) {
  String dts =
    String(year(ut)) +
    "-" +
		twoDigits(String(month(ut))) +
    "-" +
		twoDigits(String(day(ut))) +
    " - " +
		twoDigits(String(hour(ut))) +
    ":" +
		twoDigits(String(minute(ut))) +
    ":" +
		twoDigits(String(second(ut)));
  return dts;
}

void setup(void) {
  Serial.begin(57600);
  while (!Serial) {
    ;
  }
  long UnixTime = 1591107846;
 String dateTime = getDateTimeString( UnixTime );
  Serial.println( dateTime );
}

void loop() {
  // put your main code here, to run repeatedly:

}
// Output:   2020-06-02 - 14:24:06
// Sketch: 4354 bytes on Arduino Uno
// Global vars: 231 bytes on Arduino Uno
// Variant 5, using TimeLib.h and char
#include <TimeLib.h>

// Function to always return Two Digits for Date and Time using Char
char * twoDigitsChar(int digits) {
  static char digitsChar[3];
  char nul[2];
  char inpt[3];
  strcpy(nul, "0");
  itoa(digits, inpt, 10 );
  if ( digits < 10 ) {
    strcat( nul, inpt );
    strcpy(digitsChar, nul);
  } else {
    strcpy(digitsChar, inpt);
  }
  return digitsChar;
}

// Function to convert a UnixTime value to a formatted Char using TimeLib.h
char * getDateTimeChar( time_t ut ) {
  static char dtc[22];
  char yrc[5];
  strcat( dtc, itoa(year(ut), yrc, 10) );
  strcat( dtc, "-" );
  strcat( dtc, twoDigitsChar(month(ut)) );
  strcat( dtc, "-" );
  strcat( dtc, twoDigitsChar(day(ut)) );
  strcat( dtc, " - " );
  strcat( dtc, twoDigitsChar(hour(ut)) );
  strcat( dtc, ":" );
  strcat( dtc, twoDigitsChar(minute(ut)) );
  strcat( dtc, ":" );
  strcat( dtc, twoDigitsChar(second(ut)) );
  return dtc;
}

void setup(void) {
  Serial.begin(57600);
  while (!Serial) {
    ;
  }
  long UnixTime = 1591107846;
  char * dateTimeChar = getDateTimeChar( UnixTime );
  Serial.println( dateTimeChar );
}

void loop() {
  // put your main code here, to run repeatedly:

}
// Output:   2020-06-02 - 14:24:06
// Sketch: 2454 bytes on Arduino Uno
// Global vars: 246 bytes on Arduino Uno

It is clear from these examples that I needed an extra function to assure having always two digits for month, day, hour etc in the resulting output, because the standard TimeLib.h output does not provide this as far as I can see it.
So it is clear from these examples that for using the TimeLib.h library I need a lot more lines of code. But it looks like this does not have a negative effect on the finally required resources?

In the end all five variants work for me, but since I consider myself still to be a novice in the Arduino world, I would very much appreciate to get any comments on this.
Please let me know any improvements that you might have, and which variant is best, especially resource-wise. Or you might even have another better variant?
My current take is that variant 5 is the best choice because it needs the least resources and does not need extra includes in my case. Do you agree?

Cheers,
Ed