Odd problem with NTP time and ESP32

Projects Programming
21

I had the version with

configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);

running over night and it is still drifting.

Though the sketch shows another approach how to solve it.
My demo-sketch uses a variable that is located in the RTC-RAM "bootcounter"

You are using light sleep because you want to continue in your code.
With storing whatever information about what your code is doing at the moment the code goes to deepsleep in RTC-RAM variables

  • going into deepsleep is an action that your code performs at a well defined place -

allows to branch into different paths inside your code if the code comes back from deepsleep.

Coming back from deepsleep means to do a reset and coming from a reset makes sure to really sync with network-time.

If you store all the information you need for "seemless" continueing into RTC-RAM-variables you can do the reset each time.

best regards Stefan

22

Stefan. Thanks for all your input and the new thread. The time drift while asleep is something I can cope with. I understand a low power state may have an inaccurate / different clock. But I wanted believable time stamps for my data.

However the issue is becoming a bit more clear to me. I think... It may well be ESP32 specific.
This is in full an example sketch from the Arduino IDE.

#include <WiFi.h>
#include "time.h"
#include "sntp.h"

const char* ssid       = "YOUR_SSID";
const char* password   = "YOUR_PASS";

const char* ntpServer1 = "pool.ntp.org";
const char* ntpServer2 = "time.nist.gov";
const long  gmtOffset_sec = 3600;
const int   daylightOffset_sec = 3600;

const char* time_zone = "CET-1CEST,M3.5.0,M10.5.0/3";  // TimeZone rule for Europe/Rome including daylight adjustment rules (optional)

void printLocalTime()
{
  struct tm timeinfo;
  if(!getLocalTime(&timeinfo)){
    Serial.println("No time available (yet)");
    return;
  }
  Serial.println(&timeinfo, "%A, %B %d %Y %H:%M:%S");
}

// Callback function (get's called when time adjusts via NTP)
void timeavailable(struct timeval *t)
{
  Serial.println("Got time adjustment from NTP!");
  printLocalTime();
}

void setup()
{
  Serial.begin(115200);

  // set notification call-back function
  sntp_set_time_sync_notification_cb( timeavailable );

  /**
   * NTP server address could be aquired via DHCP,
   *
   * NOTE: This call should be made BEFORE esp32 aquires IP address via DHCP,
   * otherwise SNTP option 42 would be rejected by default.
   * NOTE: configTime() function call if made AFTER DHCP-client run
   * will OVERRIDE aquired NTP server address
   */
  sntp_servermode_dhcp(1);    // (optional)

  /**
   * This will set configured ntp servers and constant TimeZone/daylightOffset
   * should be OK if your time zone does not need to adjust daylightOffset twice a year,
   * in such a case time adjustment won't be handled automagicaly.
   */
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer1, ntpServer2);

  /**
   * A more convenient approach to handle TimeZones with daylightOffset 
   * would be to specify a environmnet variable with TimeZone definition including daylight adjustmnet rules.
   * A list of rules for your zone could be obtained from https://github.com/esp8266/Arduino/blob/master/cores/esp8266/TZ.h
   */
  //configTzTime(time_zone, ntpServer1, ntpServer2);

  //connect to WiFi
  Serial.printf("Connecting to %s ", ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }
  Serial.println(" CONNECTED");

}

void loop()
{
  delay(5000);
  printLocalTime();     // it will take some time to sync time :)
}

The call to configtime is made just once in the setup. BUT there is an unpredictable delay while the test within

printLocalTime();     // it will take some time to sync time :)

is running before time data is reported. The

if(!getLocalTime(&timeinfo)){
    Serial.println("No time available (yet)");

getLocalTime(&timeinfo) Seems to report immediately whether the struct &timeinfo is plausible i.e. has (ever) been populated and NOT as I assumed that an immediate time synchronisation has been done and completed correctly and the RTC clock is OK and adjusted now.

I suspect my sketch was falling into light sleep before the clock was actually updated and so I continued to see the drift in the inaccurate RTC.

This was the bit that counter intuitively did NOT get called because my sketch had gone into a light sleep.

// Callback function (get's called when time adjusts via NTP)
void timeavailable(struct timeval *t)
{
  Serial.println("Got time adjustment from NTP!");
  printLocalTime();
}

I think I can adapt this to my advantage by waiting for a call to this to report and then be confident the clock has been adjusted.

This command is under discussed, but it did not help me fully due to the issues above.

sntp_set_sync_mode(SNTP_SYNC_MODE_IMMED) ; // - Set the sync mode of the system time. It has two mode:

I want to do pulse counting while asleep and keeping the program state in light sleep is the way I thought might optimise this.
Thanks 12TA

23

It seems the callback function TimeAvailable is only called ONCE per each configtime call and not once each time getlocaltime call is made to update the clock.
So in my loop I need to call configtime periodically AND then monitor / ensure the callback function has been passed through to indicate the RTC time HAS actually been synchronised with the NTP time server call.
It was not clear to me before exactly what each function does and how.
I think I understand that the RTC synchronisation takes place a) in parallel to the main sketch code and b) after a 'random' few seconds pause which a sleep event can thwart.
In a non-sleeping sketch the ESP32 crystal controlled RTC probably does not have a need to be frequently re-synchonised to a NTP standard, but in a sleeping sketch the drift can be very significant.
12TA

25

Hi @12ta (Paul),

I have asked another question related to deepsleep-mode here
and user @noiasca has taken the effor to make it work
RE-syncing with NTP works now each time the ESP32 wakes up after deepslep.
I guess this will work too from lightsleep
here is the link to the thread.

Next step to test it with lightsleep and with looong sleeptimes

best regards Stefan

26

I have a plan. The function getlocaltime(&timeinfo) returns a boolean if the time was ever set. So, after the initial setup when it gets set OK, after a significant sleep with a probable time drift, I plan to void it somehow and return it to a blank state. I hope then it can be tested accurately to reveal an NTP mediated resynchonisation after waking from sleep. That would give easily explicable code and show to others coding the ESP32 the undocumented trap I fell into. I will test this later...
In summary? It may be best to void &timeinfo after sleeping and a potential RTC time drift.
(Sorry this approach did not work vide infra...)
12TA

27

noiasca's code does it. After a deepsleep.
What results did you get using lightsleep?

28

It seems that getlocaltime(&timeinfo, timeout) just checks the RTC for a plausible time within a timeout number of milliseconds and fills the timeinfo variable with the RTC contents. It does not in itself synchronise by NTP and can report a boolean on whether the clock was ever set.
So, it seems to me, if you want a subsequent NTP check you have to wait for the timeavailable callback be triggered by a configtime( ) call and test in that to tell you the RTC has been updated with a NTP time synch.
With a battery powered project there will be a greater current consumption during the on time while waiting for each synch, with that taking a variable time of a few seconds.
12TA

29

That was already covered in the other thread.
Why does it surprise you that connecting to the WiFi and then an NTP server will take some time?

No that is not the way. See my code in this thread.

30

What I think I have learned working with time on the ESP32...
These are my distilled conclusions.
The RTC clock is not very accurate while light_sleep is active and may drift several seconds per hour. I needed to resynchronise intermittently via a NTP call.

The code fragment...

const long  gmtOffset_sec = 0;
const int   daylightOffset_sec = 3600; // 1 hour
const char* ntpServer2 = "pool.ntp.org"; // for example
const char* ntpServer2 = "xxxxx"; // Room for two...
const char* posix_time_zone_info = "GMT0BST,M3.5.0/1,M10.5.0" ;  // Europe/London 
setenv ("TZ", posix_time_zone_info, 1);
tzset();
configTzTime (posix_time_zone_info, ntpServer1, ntpServer2); 
  // Edited to remove my typo -> configTime(gmtOffset_sec, daylightOffset_sec, ntpServer1, ntpServer2);
sntp_set_time_sync_notification_cb( timeavailable ); // // Attaches a callback function timeavailable
sntp_set_sync_mode(SNTP_SYNC_MODE_IMMED) ; // - Set the sync mode of the system time. It has two mode:

This enables an automatically immediate (i.e. not smeared) updating including automatic Daylight Savings Time / BST and time zone adjustment from a NTP server call (which is asynchronous).
It also generates a timeavailable callback function just ONCE when / if the time changes after a NTP call. So (below was edited to remove my typo)...

configTzTime (posix_time_zone_info, ntpServer1, ntpServer2);

needs to be called each time you want to update the RTC from the NTP servers.

Also importantly, to get the time into a variable use...

struct tm timeinfo; // Predefined struct designed to hold a decoded time record.
{
  bool result = getLocalTime (&timeinfo), timeout) // 5000 is the default timeout
}

See there may be 2 parameters.
N.B. This call returns false if the time was NEVER set. It will wait upto the optional timeout (default is 5000ms I think) and then only return true if either the time is plausible i.e. the RTC has been set at least once and so the RTC may or it may NOT have just been accurately set. If never set it returns false.
The actual WiFi NTP server call seems to happen in parallel with the main sketch code and takes a very variable amount of time in my case up to 2 seconds before the callback. To flag if it has happened and updated the RTC from NTP, do this from within the timeavailable callback function.
To freshen up timeinfo variable immediately from the RTC use...

getLocalTime (&timeinfo), 0) ; // Immediate RTC->timeinfo

So, to arbitrarily display the immediate time as a nice FormattedTime String use...

String FormattedTime = FormatTime() ;   // Date and time in order of significance, so can be sorted on this field. YYYY/MM/DD HH/mm/SS

String FormatTime(void)  // Format time in order of significance and constant width to aid indexing and ordering. YYYY/MM/DD HH/mm/SS
{ // https://cplusplus.com/reference/ctime/strftime/
  char RetVal[21];
  getLocalTime(&timeinfo , 0 ); // Get time from RTC before returning string. NOTE SHORT TIMEOUT as don't want to check NTP here, just get from RTC!
  strftime(RetVal, 21, "%y/%m/%d %H:%M:%S", &timeinfo); // Format the string to return
  return (RetVal);
}  // FormatTime

I hope this helps someone else see what the issues can be, as things did not seem intuitive to me or well explained from the various code fragments and examples I found. I hope this is not too long or pedantic.
12TA

31

NO!
That call sets the offset and ntp servers. Only needs to be called once. After that NTP time updates in the background with no further action on the part of the application code. You only need to call it again after waking up from sleep.

Better yet, don't call it at all. Use the version that takes the TZ variable instead:

extern "C" void configTzTime(const char* tz,
        const char* server1, const char* server2 = nullptr, const char* server3 = nullptr);
32

I'd use the standard localtime() function instead.

33

Please expand and explain further your last 2 posts. If you have time.
12TA

34

Maybe I misunderstood what you meant here. My point was once the RTC is synched to the NTP, there's never a reason to do it again unless there's a WiFi disconnect / reconnect event. As long as WiFi is connected, synching will take place periodically based on the value you provide to 'sntp_set_sync_interval()'. So don't attempt to manually resynch as long as you're connected. It's probably best to use 'SNTP_SYNC_MODE_IMMED' to synch after WiFi connect. Then switch to 'SNTP_SYNC_MODE_SMOOTH'. That way all further adjustments will be smooth. More importantly, the time value will always be monotonic.

Once synch is established, you don't need to worry about it anymore. So don't bother with that. Just use localtime(). It's one of the standard Unix Time functions that have been around for decades.

That code is counter productive. You set the TZ variable and then immediately overwrite it with the call to 'configTime()'. Let the API do the work for you. Just do:

const char* posix_time_zone_info = "GMT0BST,M3.5.0/1,M10.5.0" ;  // Europe/London 
configTzTime(posix_time_zone_info, daylightOffset_sec, ntpServer1, ntpServer2);

There's a very good discussion of Time Zones, the problems with using 'gmtOffset_sec', and NTP synching starting at Post #6 of this Thread. That one happens to be about ESP8266, but many of the concepts are applicable to ESP32.

The code below incorporates the above items. It performs an NTP synch only when the WiFi connects (either after boot up or after a WiFi disconnect / reconnect event). It contains lots of debug prints so you can see the gory details of what's going on. To see all the info logged, set the Core Debug Level in the Arduino IDE to "Debug":

image
image633×683 20 KB

#include "Arduino.h"
#include <WiFi.h>
#include "esp_sntp.h"

void handleWifiEvent(arduino_event_id_t event, arduino_event_info_t info);
void printTimeTask(void *pvParameters);
void synchTimeTask(void *pvParameters);

const EventBits_t startTimePrinting = 1UL << 0;
const EventBits_t suspendTimePrinting = 1UL << 1;
const EventBits_t synchTime = 1UL << 2;
EventGroupHandle_t controlGroupHandle;

void setup() {
  const char ssid[] = "XXXX";
  const char password[] = "XXXX";

  uint8_t retryTest = 0;

  Serial.begin(115200);
  vTaskDelay(5000);

  controlGroupHandle = xEventGroupCreate();
  if (controlGroupHandle == NULL) {
    log_e("Failed to create Control Event Group");
    return;
  }

  BaseType_t returnCode = xTaskCreatePinnedToCore(synchTimeTask, "Synch Time Task", 4000, NULL, 3, NULL, CONFIG_ARDUINO_RUNNING_CORE);
  if (returnCode != pdTRUE) {
    log_e("Failed to create Synch Time Task");
    return;
  }

  returnCode = xTaskCreatePinnedToCore(printTimeTask, "Print Task", 1700, NULL, 5, NULL, CONFIG_ARDUINO_RUNNING_CORE);
  if (returnCode != pdTRUE) {
    log_e("Failed to create Print Task");
    return;
  }

  log_i("Station MAC Address: %s", WiFi.macAddress().c_str());
  log_i("Connecting to %s", ssid);
  WiFi.onEvent(handleWifiEvent);

  while (WiFi.status() != WL_CONNECTED) {
    if (retryTest == 0) {
      WiFi.disconnect();
      WiFi.begin(ssid, password);
      retryTest = 5;
    } else {
      log_d(".");
      retryTest--;
      vTaskDelay(1000);
    }
  }
}

void loop() {
}

void synchTimeTask(void *pvParameters) {
  bool firstTime = true;
  const char ntpServer1[] = "time.nist.gov";
  const char ntpServer2[] = "pool.ntp.org";
  const uint32_t syncInterval = 3600UL * 1000;
  uint8_t retryTest = 0;
  uint8_t attemptsBeforeReset = 5;
  timeval tv;
  tm *timeinfo;
  char timeString[100];
  time_t now;

  for (;;) {
    xEventGroupWaitBits(controlGroupHandle, synchTime, pdTRUE, pdFALSE, portMAX_DELAY);

    if (!firstTime) {
      xEventGroupSetBits(controlGroupHandle, suspendTimePrinting);
    }
    firstTime = false;

    //init and get the time
    sntp_set_sync_mode(SNTP_SYNC_MODE_IMMED);  // Set for immediate synch on WiFi connect
    sntp_set_sync_interval(syncInterval);
    configTzTime("EST5EDT,M3.2.0,M11.1.0", ntpServer1, ntpServer2);

    retryTest = 0;
    while (true) {                 // Wait for NTP sync to take effect
      if (retryTest == 0) {
        if ((--attemptsBeforeReset) == 0) {
          ESP.restart();
        }
        log_i("Waiting for Time Sync");
        sntp_sync_time(&tv);
        retryTest = 10;
      } else {
        log_d(".");
        retryTest--;
      }
      vTaskDelay(2000);
      now = time(nullptr);
      timeinfo = localtime(&now);
      if (timeinfo->tm_year >= (2022 - 1900)) {
        break;
      }
    }

    sntp_set_sync_mode(SNTP_SYNC_MODE_SMOOTH);  // Switch to smooth synch
    sntp_sync_status_t syncStatus = sntp_get_sync_status();
    switch (syncStatus) {
      case SNTP_SYNC_STATUS_RESET:
        log_d("SNTP_SYNC_STATUS_RESET");
        break;

      case SNTP_SYNC_STATUS_COMPLETED:
        log_d("SNTP_SYNC_STATUS_COMPLETED");
        break;

      case SNTP_SYNC_STATUS_IN_PROGRESS:
        log_d("SNTP_SYNC_STATUS_IN_PROGRESS");
        break;

      default:
        log_e("Unknown Sync Status");
        break;
    }

    sntp_sync_mode_t mode = sntp_get_sync_mode();
    switch (mode) {
      case SNTP_SYNC_MODE_IMMED:
        log_d("SNTP_SYNC_MODE_IMMED");
        break;

      case SNTP_SYNC_MODE_SMOOTH:
        log_d("SNTP_SYNC_MODE_SMOOTH");
        break;

      default:
        log_e("Unknown Sync Mode");
        break;
    }

    uint32_t interval = sntp_get_sync_interval();
    log_d("NTP Sync Interval = %d", interval);

    timeinfo = gmtime(&now);
    log_d("UTC Info:");
    log_d("Year = %d, Mon = %d, Day = %d", timeinfo->tm_year, timeinfo->tm_mon, timeinfo->tm_mday);
    log_d("Hour = %d, Min = %d, Sec = %d", timeinfo->tm_hour, timeinfo->tm_min, timeinfo->tm_sec);
    log_d("DoY = %d, DoW = %d, IsDST = %d", timeinfo->tm_yday, timeinfo->tm_wday, timeinfo->tm_isdst);
    log_d("TZ = %s", getenv("TZ"));

    xEventGroupSetBits(controlGroupHandle, startTimePrinting);
  }
}

void printTimeTask(void *pvParameters) {
  const TickType_t delayTime = 1000;
  char timeString[100];
  TickType_t wakeTime;
  bool suspend;
  EventBits_t testSuspend;

  for (;;) {
    xEventGroupWaitBits(controlGroupHandle, startTimePrinting, pdTRUE, pdFALSE, portMAX_DELAY);
    wakeTime = xTaskGetTickCount();
    suspend = false;
    while (!suspend) {
      time_t now = time(nullptr);
      tm *timeinfo = localtime(&now);
      strftime(timeString, 100, "Local Time: %A, %B %d %Y %H:%M:%S %Z", timeinfo);
      //log_i("%s", timeString);
      Serial.println(timeString);
      xTaskDelayUntil(&wakeTime, delayTime);
      testSuspend = xEventGroupWaitBits(controlGroupHandle, suspendTimePrinting, pdTRUE, pdFALSE, 0);
      if ((testSuspend & suspendTimePrinting) != 0) {
        suspend = true;
      }
    }
  }
}

void IRAM_ATTR handleWifiEvent(arduino_event_id_t event_id, arduino_event_info_t info) {

  switch (event_id) {
    case ARDUINO_EVENT_WIFI_READY:
      log_d("WiFi Ready");
      break;

    case ARDUINO_EVENT_WIFI_STA_START:
      log_d("WiFi Start");
      break;

    case ARDUINO_EVENT_WIFI_STA_STOP:
      log_d("WiFi Stop");
      break;

    case ARDUINO_EVENT_WIFI_STA_CONNECTED:
      log_d("Connected to SSID: %s", reinterpret_cast<char*>(info.wifi_sta_connected.ssid));
      break;

    case ARDUINO_EVENT_WIFI_STA_GOT_IP: {
        uint8_t bytes[4];
        uint32_t ipAddress = info.got_ip.ip_info.ip.addr;
        for (uint8_t i = 0; i < 4; i++) {
          bytes[i] = ipAddress & 0xFF;
          ipAddress >>= 8;
        }
        log_d("Got IP Address: %d.%d.%d.%d", bytes[0], bytes[1], bytes[2], bytes[3]);
        xEventGroupSetBits(controlGroupHandle, synchTime);
        break;
      }

    case ARDUINO_EVENT_WIFI_STA_LOST_IP:
      log_d("IP Address Lost");
      break;

    case ARDUINO_EVENT_WIFI_STA_DISCONNECTED:
      log_d("Disconnected From SSID: %s, Reason: %d", reinterpret_cast<char*>(info.wifi_sta_disconnected.ssid),
            info.wifi_sta_disconnected.reason);
      break;

    default:
      log_d("WiFi Event: %d", static_cast<uint8_t>(event_id));
      break;
  }
}
35

Sorry. I thought I had made it very clear earlier on as my sketch us battery powered so sleeps for 99% of the time with WiFi not connected and in light_sleep. The RTC drifts about 300ms per minute while asleep which needs correcting every x loops to keep the RTC time vaguely in synch.
I will edit to correct to your suggestion, I had had a case-typo in the code and so it would not compile configTzTime does.

configTzTime (posix_time_zone_info, ntpServer1, ntpServer2);

Thank You!
12TA

36

One thing to keep in mind is that the code for the ESP32 and the ESP8266 are different.
The code is maintained by different developers and the differences are frustratingly different.
There are differences in the SNTP code implementation, the configTime() API, the way callbacks work I as well as the syncing behaviors.
The wall callbacks for time setting works is very different on the two platforms.

So anything you may determine for one platform may not apply or work for the other.

Also depending on when you look at the code, the differences and issues can vary as the code on each is occasionally updated.
From interacting with developers on both platforms, I can say that the ESP8266 developers are much more reasonable and responsive to resolving issues.

The only way to really know how it works and behaves is to actually look at the source code for the core you have installed and are using.

On the ESP8266 the offset stuff is broken and will the developers have declared that they will never fix it do to fears of breaking existing code in the field that may depend on the existing behavior.
From memory on a very recent Arduino forum thread and a subsequent test sketch I ran, some of the offset stuff on the ESP32 is not an issue due to the code implementation being different.
And in some cases things don't happen until you call yield()

But again, YMMV since things can vary by code version.

IMO, I would recommend never using the offset stuff, it really isn't needed. Just use the TZ capability (you will also gain automatic DST adjustments capability) and if using an RTC put UTC time in the RTC vs local time.
Life will be much simpler as it will keep the sketch code portable between the platforms. (well at least the code that does deal with the callbacks or the initial configTime() vs configzTime() )

--- bill

37

My comments and the code I provided cover that. As I said, it performs an NTP re-synch every time the WiFi reconnects. That happens after you wake up from sleep.

No, it only needs be be corrected once ... when the WiFi reconnects after waking up. Then you can go as many "loops" as you want without doing any more resynching. Again, the WiFi reconnect event will trigger an RTC resynch.

38

Sorry. I thought I had made it very clear earlier on as my sketch us battery powered so sleeps for 99% of the time with WiFi not connected and in light_sleep. The RTC drifts about 300ms per minute which needs correcting every x loops to keep the RTC time vaguely in synch.

39

Have you looked at using a crystal for the RTC ?

40

I was attempting a software only fudge. The precise time is not critical if I can correct drifting from NTP every so often.
Thanks
12TA

41

Which part of my comments didn't you understand? Yes, I know you need to resynch after light sleep. So every time you wake up from sleep, the WiFi will reconnect. My code uses the reconnect event to then trigger the resynch. After that happens, the RTC will remain synched until you go to sleep again.

Then, when you wake up again, the WiFi will reconnect. My code uses the reconnect event to then trigger the resynch. After that happens, the RTC will remain synched until you go to sleep again.

Rinse and Repeat. Ad Infinitum.

I don't know how to explain it more clearly that that.