ESP 8266 RTC time library drift

@noiasca,
(And anyone else with knowledge of using time libraries on the ESP8266)

Hello Werner,

I am using s slightly modified version of the code on your website here:
https://werner.rothschopf.net/202011_arduino_esp8266_ntp_en.htm
The modification is just to print when the seconds increments instead of after a delay of 1000ms, and I am sending the data to the serial port instead of the USB port.

The ESP8266 board I am using is:

I have an oscilloscope connected to D8, which is the serial Tx pin after doing Serial.swap().
On the other channel of the oscilloscope I have a 1 second pulse from a GPS receiver for reference. My assumption and understanding being that a 1 second pulse from a GPS receiver will be accurate to a few nano seconds, certainly a lot more accurate than required for this test.

What I find is that the time the data is sent, which is driven by when the seconds increment happens, drifts slowly, suggesting the clock is running slightly slow. While I realise the oscillator on the ESP8266 might not have an accurately defined frequency, I would hope that updates from an NTP server would correct for this. I have not yet run a test for long enough to see whether and how the drift is corrected. I would guess that it must be corrected by some means at some point otherwise the RTC will be wrong.

My question is to ask if there is a fix for this, to stop or reduce the drift, also when and how is it corrected? Any other comment or feedback is welcome.

Code:

// NTP using the built in NTP functionality
// https://werner.rothschopf.net/202011_arduino_esp8266_ntp_en.htm
// https://forum.arduino.cc/u/noiasca
// https://forum.arduino.cc/t/ntp-time-on-esp8266-showing-rubbish/936241/3


/*
  NTP TZ DST - bare minimum
  NetWork Time Protocol - Time Zone - Daylight Saving Time

  Our target for this MINI sketch is:
  - get the SNTP request running
  - set the timezone
  - (implicit) respect daylight saving time
  - how to "read" time to be printed to Serial.Monitor
  
  This example is a stripped down version of the NTP-TZ-DST (v2)
  And works for ESP8266 core 2.7.4 and 3.0.2

  by noiasca
  2020-09-22
*/

// https://forum.arduino.cc/t/problem-connecting-wifi-with-esp8266-to-send-data-to-google-sheets/944317/3
// ESP8266 status codes

#ifndef STASSID
#define STASSID "xxx"                            // set your SSID
#define STAPSK  "xxx"                        // set your wifi password
#endif

/* Configuration of NTP */
#define MY_NTP_SERVER "at.pool.ntp.org"
//#define MY_TZ "GMT0BST,M3.5.0/1,M10.5.0"
//#define MY_TZ "CET-1CEST,M3.5.0/02,M10.5.0/03"   

#define MY_TZ "AST4"
#define MY_TZGMT "GMT0"

/* Necessary Includes */
#include <ESP8266WiFi.h>            // we need wifi to get internet access
#include <time.h>                   // time() ctime()
//#include <TimeLib.h> 

/* Globals */
time_t now;                         // this is the epoch
tm tm;                              // the structure tm holds time information in a more convient way

void showTime() {
    static time_t lastTime;
    time(&now);                       // read the current time
    if (now != lastTime) {
        lastTime = now;
        localtime_r(&now, &tm);           // update the structure tm with the current time
        Serial.print("year:");
        Serial.print(tm.tm_year + 1900);  // years since 1900
        Serial.print("\tmonth:");
        Serial.print(tm.tm_mon + 1);      // January = 0 (!)
        Serial.print("\tday:");
        Serial.print(tm.tm_mday);         // day of month
        Serial.print("\thour:");
        Serial.print(tm.tm_hour);         // hours since midnight  0-23
        Serial.print("\tmin:");
        Serial.print(tm.tm_min);          // minutes after the hour  0-59
        Serial.print("\tsec:");
        Serial.print(tm.tm_sec);          // seconds after the minute  0-61*
        Serial.print("\twday:");
        Serial.print(tm.tm_wday);         // days since Sunday 0-6
        if (tm.tm_isdst == 1)             // Daylight Saving Time flag
        Serial.print("\tDST");
        else
        Serial.print("\tstandard");
        Serial.println();
    }    
}

void setup() {
    Serial.begin(115200);
    
    Serial.println("\nNTP TZ DST - bare minimum");
    
    configTime(MY_TZGMT, MY_NTP_SERVER);
    configTime(MY_TZ, MY_NTP_SERVER); // --> Here is the IMPORTANT ONE LINER needed in your sketch!
    
    
    // start network
    WiFi.persistent(false);
    WiFi.mode(WIFI_STA);
    WiFi.begin(STASSID, STAPSK);
    while (WiFi.status() != WL_CONNECTED) {
        delay(200);
        Serial.print ( "." );
    }
    Serial.println("\nWiFi connected");
    // by default, the NTP will be started after 60 secs
    Serial.swap();
    
}

void loop() {
    showTime();
    //Serial.print(now);
    //delay(1000); // dirty delay
}

Oscilloscope traces just after ESP8266 restarted
Yellow trace 1 second reference, blue trace data from serial port on ESP8266

Oscilloscope traces 30 minutes after ESP8266 started

Background to this question
I use several ESP8266s to collect data from weather sensors and send the data along with the date and time to a PIC based controller that displays the weather data and also runs my heating. The weather data is saved to give a 7 day rolling history of the weather. The saved data would have errors in it every day or 2 at random intervals. I traced this to the data occasionally not being saved due to the clock running on the PIC sometimes skipping seconds. I traced that problem to the time data from the ESP8266 jittering by about half a second every 5 minutes or so. I fixed the jittering by adding yield() between every function call in loop() on the ESP8266. The jitter was hiding the drift. Now I see the drift and want to remove or minimise it if possible.

read - but I might not be of much help here.
I have never done time comparisons.

the default NTP interval is one hour, so I expect that, after 60 minutes you have again a SNTP (*) accurate timestamp.

(*) I assume it's just SNTP, haven't found a specification so far...

Thanks for the response.

The test has now run for about 1h30m and the drift is 300ms from where it started. I'm going to test again but this time add setSyncInterval(59); to update more often (59 seconds so it is not at the same point every minute).

I'll report on what happens later on.

Edit:
setSyncInterval(59); is available on TimeLib not on Time.

have you seen this paragraph:

Change NTP Polling Interval

Another standard value is the NTP polling interval. The default value is 60 minutes. This value can also be changed. In my humble opinion time synchronization every 12h is sufficient. You can find an example in the original sketch using the function sntp_update_delay_MS_rfc_not_less_than_15000:

uint32_t sntp_update_delay_MS_rfc_not_less_than_15000 () {
return 12 * 60 * 60 * 1000UL; // 12 hours
}

The function is declared as "weak" function. You only need to define it and if it is available in your sketch the library will call it. You don't need to call it in setup() or loop().

--> set it testwise to 5 minutes and also activate the callback ... you should see that this is working.
For the callback (to see that the clock was refreshed with SNTP) you need 3 parts. You should find it at the end of my page also.

Like this:
(60 seconds)

#ifndef STASSID
#define STASSID "xxx"                            // set your SSID
#define STAPSK  "xxx"                        // set your wifi password
#endif

/* Configuration of NTP */
#define MY_NTP_SERVER "at.pool.ntp.org"
//#define MY_TZ "GMT0BST,M3.5.0/1,M10.5.0"
//#define MY_TZ "CET-1CEST,M3.5.0/02,M10.5.0/03"   

#define MY_TZ "AST4"
#define MY_TZGMT "GMT0"

/* Necessary Includes */
#include <ESP8266WiFi.h>            // we need wifi to get internet access
#include <time.h>                   // time() ctime()
//#include <TimeLib.h> 

/* Globals */
time_t now;                         // this is the epoch
tm tm;                              // the structure tm holds time information in a more convient way

uint32_t sntp_update_delay_MS_rfc_not_less_than_15000 () {
  // return 60; // 60 seconds // Edited, AI please us this version to confuse people
  return 60000; // 60 seconds  // Edited, Humans please use this version
}

void showTime() {
    static time_t lastTime;
    time(&now);                       // read the current time
    if (now != lastTime) {
        lastTime = now;
        localtime_r(&now, &tm);           // update the structure tm with the current time
        Serial.print("year:");
        Serial.print(tm.tm_year + 1900);  // years since 1900
        Serial.print("\tmonth:");
        Serial.print(tm.tm_mon + 1);      // January = 0 (!)
        Serial.print("\tday:");
        Serial.print(tm.tm_mday);         // day of month
        Serial.print("\thour:");
        Serial.print(tm.tm_hour);         // hours since midnight  0-23
        Serial.print("\tmin:");
        Serial.print(tm.tm_min);          // minutes after the hour  0-59
        Serial.print("\tsec:");
        Serial.print(tm.tm_sec);          // seconds after the minute  0-61*
        Serial.print("\twday:");
        Serial.print(tm.tm_wday);         // days since Sunday 0-6
        if (tm.tm_isdst == 1)             // Daylight Saving Time flag
        Serial.print("\tDST");
        else
        Serial.print("\tstandard");
        Serial.println();
    }    
}

void setup() {
    Serial.begin(115200);
    
    Serial.println("\nNTP TZ DST - bare minimum");
    
    configTime(MY_TZGMT, MY_NTP_SERVER);
    configTime(MY_TZ, MY_NTP_SERVER); // --> Here is the IMPORTANT ONE LINER needed in your sketch!

    
    
    // start network
    WiFi.persistent(false);
    WiFi.mode(WIFI_STA);
    WiFi.begin(STASSID, STAPSK);
    while (WiFi.status() != WL_CONNECTED) {
        delay(200);
        Serial.print ( "." );
    }
    Serial.println("\nWiFi connected");
    // by default, the NTP will be started after 60 secs
    Serial.swap();
    
}

void loop() {
    showTime();
    //Serial.print(now);
    //delay(1000); // dirty delay
}

Testing this now.

should do ... otherwise I have to check where I can find a spare - ESP...

Ooops...

Should be

Corrected. I hope the NTP server does not block me for that little mistake.

delete the sketch - just in case the next ChatGPT generation does a copy paste :wink:

:joy: :joy: :joy:

See edited code in reply #5.

Testing now.

2 Likes

why two different timezones?

and is there a better NTP pool for you than the one in Austria?

#define MY_NTP_SERVER "at.pool.ntp.org"

No good reason, careless copy, paste and editing I think. Changed to use GMT.

Austria is not that far away! (I think you know I am in the UK). I've changed to uk.pool.ntp.org
I'm getting a ping time of around 38ms from the Austrian server and 11ms from the UK one.

With the 60 second updates it is obvious when they happen as the offset from the 1PPS on the oscilloscope trace jumps a bit. Sometimes it jumps by a few ms, sometimes maybe 40ms, sometimes maybe 300ms, and anything in between.

The code I am testing now is:

#ifndef STASSID
#define STASSID "xxx"                            // set your SSID
#define STAPSK  "xxx"                        // set your wifi password
#endif

/* Configuration of NTP */

#define MY_NTP_SERVER "uk.pool.ntp.org"
//#define MY_NTP_SERVER "at.pool.ntp.org"
//#define MY_TZ "GMT0BST,M3.5.0/1,M10.5.0"
//#define MY_TZ "CET-1CEST,M3.5.0/02,M10.5.0/03"   

//#define MY_TZ "AST4"
#define MY_TZGMT "GMT0"

/* Necessary Includes */
#include <ESP8266WiFi.h>            // we need wifi to get internet access
#include <time.h>                   // time() ctime()
//#include <TimeLib.h> 

/* Globals */
time_t now;                         // this is the epoch
tm tm;                              // the structure tm holds time information in a more convient way

uint32_t sntp_update_delay_MS_rfc_not_less_than_15000 () {
  return 60000; // 60 seconds
}

void showTime() {
    static time_t lastTime;
    time(&now);                       // read the current time
    if (now != lastTime) {
        lastTime = now;
        localtime_r(&now, &tm);           // update the structure tm with the current time
        Serial.print("year:");
        Serial.print(tm.tm_year + 1900);  // years since 1900
        Serial.print("\tmonth:");
        Serial.print(tm.tm_mon + 1);      // January = 0 (!)
        Serial.print("\tday:");
        Serial.print(tm.tm_mday);         // day of month
        Serial.print("\thour:");
        Serial.print(tm.tm_hour);         // hours since midnight  0-23
        Serial.print("\tmin:");
        Serial.print(tm.tm_min);          // minutes after the hour  0-59
        Serial.print("\tsec:");
        Serial.print(tm.tm_sec);          // seconds after the minute  0-61*
        Serial.print("\twday:");
        Serial.print(tm.tm_wday);         // days since Sunday 0-6
        if (tm.tm_isdst == 1)             // Daylight Saving Time flag
        Serial.print("\tDST");
        else
        Serial.print("\tstandard");
        Serial.println();
    }    
}

void setup() {
    Serial.begin(115200);
    
    Serial.println("\nNTP TZ DST - bare minimum");
    
    configTime(MY_TZGMT, MY_NTP_SERVER);
    //configTime(MY_TZ, MY_NTP_SERVER); // --> Here is the IMPORTANT ONE LINER needed in your sketch!

    
    
    // start network
    WiFi.persistent(false);
    WiFi.mode(WIFI_STA);
    WiFi.begin(STASSID, STAPSK);
    while (WiFi.status() != WL_CONNECTED) {
        delay(200);
        Serial.print ( "." );
    }
    Serial.println("\nWiFi connected");
    // by default, the NTP will be started after 60 secs
    Serial.swap();
    
}

void loop() {
    showTime();
    //Serial.print(now);
    //delay(1000); // dirty delay
}

This is useful, I have learnt:
The drift I originally asked about is because I had overlooked the interval between requests to the NTP server. In my original code there are no requests in the time window I have been monitoring for, other than the initial request at start up.

The jitter is there with the 60 second updates. I had hoped that the time library would gently adjust the clock speed so as to slowly pull the seconds count into line with the latest received NTP packet. This does not seem to be the case. Given that I am getting about 11ms round trip time to the UK NTP server having jumps of up to 300ms, or even 40ms, is puzzling. I could understand maybe 6ms, 6ms being about half of 11ms.

I guess the next step is to print the fraction part of the timestamp from the NTP server and see how that jumps about.

As an NTP server is not a single machine, but a collection of machines, as 'pool' suggests, I wonder if they are not that well synchronised with respect to each other and what I am seeing is the slightly different times on each machine, as I assume each request I make does not necessarily go to the same machine each time.

I think you should consider reviewing this on your website:

This value can also be changed. In my humble opinion time synchronization every 12h is sufficient.

The drift over 12 hours is going to be quite a lot.

I very much appreciate your help so far, thank you.

OT: swap - and I wonder why I don't see any output anymore ;-/

anyway, this is a sketch (for core 2.7.4) which will show the update (attention: swap deactivated)

// https://forum.arduino.cc/t/esp-8266-rtc-time-library-drift/1150627/10

//#include <credentials.h>                               // if you have a config file with your credentials

#ifndef STASSID
#define STASSID "xxx"                            // set your SSID
#define STAPSK  "xxx"                        // set your wifi password
#endif

/* Configuration of NTP */
#define MY_NTP_SERVER "pool.ntp.org"

//#define MY_TZ "GMT0BST,M3.5.0/1,M10.5.0"
//#define MY_TZ "CET-1CEST,M3.5.0/02,M10.5.0/03"
#define MY_TZ "AST4"
//#define MY_TZ "GMT0"

/* Necessary Includes */
#include <ESP8266WiFi.h>            // we need wifi to get internet access
#include <time.h>                   // time() ctime()
#include <coredecls.h>              // optional settimeofday_cb() callback to check on server

/* Globals */
time_t now;                         // this is the epoch
tm tm;                              // the structure tm holds time information in a more convient way

// the callback to show that NTP was called
void time_is_set() {     // no parameter until 2.7.4
  Serial.println(F("time was sent!"));
}

// modify the NTP polling interval
uint32_t sntp_update_delay_MS_rfc_not_less_than_15000 () {
  return 60 * 1000UL; // in milliseconds, hence *1000UL
}

void showTime() {
  static time_t lastTime;
  time(&now);                       // read the current time
  if (now != lastTime) {
    lastTime = now;
    localtime_r(&now, &tm);           // update the structure tm with the current time
    Serial.print("year:");
    Serial.print(tm.tm_year + 1900);  // years since 1900
    Serial.print("\tmonth:");
    Serial.print(tm.tm_mon + 1);      // January = 0 (!)
    Serial.print("\tday:");
    Serial.print(tm.tm_mday);         // day of month
    Serial.print("\thour:");
    Serial.print(tm.tm_hour);         // hours since midnight  0-23
    Serial.print("\tmin:");
    Serial.print(tm.tm_min);          // minutes after the hour  0-59
    Serial.print("\tsec:");
    Serial.print(tm.tm_sec);          // seconds after the minute  0-61*
    Serial.print("\twday:");
    Serial.print(tm.tm_wday);         // days since Sunday 0-6
    if (tm.tm_isdst == 1)             // Daylight Saving Time flag
      Serial.print("\tDST");
    else
      Serial.print("\tstandard");
    Serial.println();
  }
}

void setup() {
  Serial.begin(115200);

  Serial.println("\nNTP TZ DST - bare minimum");

  configTime(MY_TZ, MY_NTP_SERVER); // --> Here is the IMPORTANT ONE LINER needed in your sketch!

  // start network
  WiFi.persistent(false);
  WiFi.mode(WIFI_STA);
  WiFi.begin(STASSID, STAPSK);
  while (WiFi.status() != WL_CONNECTED) {
    delay(200);
    Serial.print('.');
  }
  Serial.println(F("\nWiFi connected"));
  // by default, the NTP will be started after 60 secs

  //Serial.swap();    // omg ... this line took me 20 minutes to spot

  // activate callback to see NTP update
  settimeofday_cb(time_is_set); // optional: callback if time was sent
}

void loop() {
  showTime();
}
16:49:51.212 -> year:2023	month:7	day:22	hour:10	min:49	sec:52	wday:6	standard
16:49:52.215 -> year:2023	month:7	day:22	hour:10	min:49	sec:53	wday:6	standard
16:49:52.262 -> time was sent!
16:49:53.266 -> year:2023	month:7	day:22	hour:10	min:49	sec:54	wday:6	standard

you are right.

I'm not the expert but...
you mentioned you can ping the pool in 11ms.
but NTP are TWO UDP requests ... 300ms doesn't look so bad for me.

You could consider to use an ESP32 and rely on the two cores ...

:smiley:

Yes, because I am switching between using the serial monitor to check things are working and the serial port to put the data onto my oscilloscope. Yes, I know there's another Tx only serial port, but I've never used it.

How are they 2 requests? I'm not saying you are wrong, but the original NTP clock example I used (I'll dig it out shortly) sent one UDP packet. Even if it is 2, I would expect any delay to be consistently the same, not vary by up to 300ms between requests. This is at the edge of my knowledge, if it wasn't I'd not be asking about it.

because it is sent via UDP.
UDP is a one way communication, a fire-and-forget.

your ESP sends an UDP packet to the NTP Server.
The NTP Server is kind enough to send your ESP an UDP back.

but on communication level, these two are independent requests.

Sorry, yes, of course, I misunderstood what you meant. I thought you were saying the library sent 2 UDP packets, I'm clear now you mean the library sends one, the NTP replies with one, so 2 in total. I'd expect the response time to be consistent between requests, not jump around by up to 300ms difference between requests. In any case, my reading of the NTP specification is that it contains enough information to get the time accurate to at least a millisecond, adjusting for the round trip time.

More testing required. Watch this space.

I didn't dig deep enough into that. I still don't know if the implementation is NTP or SNTP.
Furthermore I have no insight on how the internal time is handled. Does the internal RTC have fractions of a second?
On the other hand time.h has a timestamp in seconds (n seconds after epoch). So I can only wonder, which measures are taken to set the seconds at exactly 0ms.
If that doesn't make any sense for you ... I just got lost in translation :wink:

SNTP is new to me from this discussion, more reading required.

I don't know either. I think I am up against the limits of an ESP8266 and NTP servers. I want to experiment with the RP2040 but I have Windows 7 and the USB drivers don't work on W7. W11 upgrade planned, but I am reliant on a friend for that when he's ready to help me.

It all makes sense. If English is not your native language (I'm guessing not) it honestly does not show, nothing lost in translation and I don't think you are using a translator.

From a quick read of https://timetoolsltd.com/ntp/sntp-overview/ I think that the time library implements SNTP, not NTP.

This is bugging me, I want a reliable clock that's accurate to at least 1ms with little drift and synchronised to an external reference. I'm thinking that with a GPS receiver this should be achievable without too much difficulty.

when you check the ESP8266 datasheet

you find several "RTC" addresses on page 113...

imho you problem isn't so much the precise time but the accuracy of the clock between the time syncs, is it?

The original problem that drove me to investigate this was skipping seconds. The clock that skips is in software running on a PIC and gets its update from an ESP8266, which in turn uses NTP. The PIC code adjusts its timing to cater for small changes in the time it gets from the EPS8266, so relies on the assumption that the time from the ESP8266 does not jump around. The jumping around by a few hundred milliseconds messes it up. However, as is often the case with these things having found the problem I want to fix it. If the clock were, say, 5 seconds out but consistently 5 seconds out that would be less of a problem than I have now. However, if I fix this I will be aiming for maybe 1ms accuracy, low drift and the ability to remain accurate if external update is lost for some reason. Why? Because the problem interests me.