Advice on using NTP time with ESP8266?

I have asked here before concerning usage of the NTP time servers to acquire accurate time via the network.
I got advice and links to an NTP library to use.

But there is one thing I am not understanding how to do properly yet:
How do I read the current time as a time_t value inside the sketch at various places without actually calling the NTP server?
The NTP object should handle repeated sync to the server as needed, so as to limit network traffic.

It looks like if I use the NTP object to get the current time in my sketch it always makes a call to the server and quite often this fails and I get a time of zero.
Is there no way to read the synced time without calling out to the server?

My current project is a device that will sleep for long times, then wake up for a short while and perform a task where it needs the time. So there is an NTP request in setup() that executes once and for further use I would like to be able to call a function that gives back the current time in a time_t format without calling the NTP server at all.

Here is how the NTP object is started in loop() after checking that WiFi has been connected so there is an Internet availability:

	void HandleNtpClient()
	{
		if (wifiFirstConnected) {  //One time execution following WiFi connect
			wifiFirstConnected = false;
			NTP.begin ("pool.ntp.org", timeZone, true, minutesTimeZone);
			NTP.setInterval (60, 86400);	//Initial NTP call repeat 1 min, then 24h between calls
		}

		if (syncEventTriggered) {
			processSyncEvent (ntpEvent);
			syncEventTriggered = false;
		}
	}

void loop()
{
	//Check if a new DHT reading is requested:
	unsigned long tickcount = millis();
	HandleNtpClient();
        ... other loop code .....
}

Then in the actual code I need to read the current time in several places, but if I use this:

t = NTP.getTime();

It looks like NTP actually calls to the server every time....
Even though NTP was successfully initialized NTP.getTime() fails often if I call it later in several places.

It seems like a function time_t CurrentTime() should exist that uses a single (NTP internal) server call plus the millis() function to calculate the current time....

And BTW:
Is there a convenient function to stuff a time_t variable content into a string (char array) in a time or date or date time format without resorting to sprintf or similar? I just want to send the time or date out in ISO format to the serial debug output. Like:
2019-02-03 08:16:01

You can keep local time with the TimeLib.h library. This runs from the processor's clock. You use NTP time from the network to periodically "true up" your local time. See TimeNTP example in the Time library and NTPClient in the ESP8266WiFi libarary.

You can keep local time with the TimeLib.h library. This runs from the processor's clock. You use NTP time from the network to periodically "true up" your local time.

My current project is a device that will sleep for long times, then wake up for a short while and perform a task where it needs the time.

What sleep mode are you using? What power usage in sleep are you looking for? How are you waking from sleep?

Is this project on an AVR processor?

  1. Deep Sleep
  2. 20 uA
  3. ESP.deepSleep(sleeptime); (with GPIO16 wired to RST)
  4. It is on an ESP-07 module using the ESP8266 WiFi processor

I found an answer to my questions now:

A) To return a calculated time value based on the latest NTP sync:

t = now();

B) To format the time_t variable in different ways as a String in ISO format:

time_t t;
tn = now();
String Msg;
Msg = NTP.getTimeStr(tn); // 00:29:51
Msg = NTP.getTimeDateString(tn); // 2019-02-04 00:29:51

if you go back to your first post here on the forum and read the first comment (by me) you can see I send you a link to example sketch which had this too. and I remember I referenced it later at least one time

https://github.com/jandrassy/TelnetStream/blob/master/examples/TelnetStreamTest/TelnetStreamTest.ino

Hi Juraj,
I guess you mean the thread Function to retrieve current time on ESP8266 which I started back in May 2018?

At that time I was working on the mains powered version of my logging system while recovering from a fall from a roof that broke my hip...

Anyway I had a look at that and found it to be a bit over my head, unfortunately.
Later I got wind of the NtpClientLib library, which also needed the TimeLib library and I got some kind of functionality running without really understanding how it worked. With the libs there was just a few lines of my own code in the sketch....

What I did not realize at the time was that apparently the NTP.begin() and NTP.setInterval() was basically all that was needed in order to get the functionality I was looking for. In fact it seems like no more calls to the NTP object such as getTime() are needed, it keeps track of time in the background itself.

But the functions I need are divided between the TimeLib and NtpClientLib libraries such that to get the current time once NTP had been initialized using the two calls above one just need to execute now(), which returns the time_t value for the current second.
It looks like NtpClientLib and TimeLib are interconnected with each other. Which I did not know/understand.

Until now....

No need to do anything else than now(), except if one wants to format the time in any good way...
In that case using the NTP object's methods for formatting time by sending in the time_t variable as an argument is how it is done.
So NTP.getTimeDateString(time_t variable) returns the wanted string with the date and time in ISO format.
There are more such NTP functions too.
And the TimeLib contains stuff like hour(time) etc to help when one wants to decode time_t variables.

So I guess I am good now, I just have to figure out how to use this in the deepsleep scenario. On my way.....

Check out the 'breakTime()' function in TimeLib library. It takes a time_t variable (really just a 'long' or int32_t) and fills in a 'tmElements_t' struct. You pass it the time_t and a pointer to the tmElements_t.

The now(), like most of the functions in the TimeLib library depend on millis(). Do you know if millis() will continue to be properly updated during your deep sleep? If not, you're gonna have to go over the net to an NTP server every time you wake up from sleep.

I will have a look at the breakTime() function, for sure.

Regarding millis(), it stops running during deep sleep. This is the reason why I want to use the network time to sync across sleep intervals. The wakeup is a device reset.
The deepsleep function takes an argument expressed in us(!) for the sleep time and I thought that this would be useful to schedule a precise time so that the measurements would happen at the set interval.
However, it turns out that the units I have checked are all 4-6% slow (so why using microseconds?).
I have tried to add a calibration value that can be configured manually to multiply the time calculated to get to the correct start time. This kind of works but there is still a drift. But is better than without.
If I instead could set the time a little short and then have the measurement start depending on the NTP time retrieved in the setup() function it could be more accurate. After all, for what I know the deepsleep oscillator could well be temperature dependent too. And then a fixed correction value will not work...

BosseB:
The wakeup is a device reset.

If that's true, won't setup() run again causing another NTP time retrieval?

I would discourage using TimeLib as it has no knowledge of local time zones or how to do DST adjustments.
You don't need any 3rd party libraries to use NTP and have proper local time, including local DST time adjustments.
Everything you need is already included in the ESP8266 core.

Attached is a simple example to show how to use NTP.

In a real system, you would likely want to use WiFi manager, and add a few custom fields like the TZ string.

--- bill

NTP-Clock-MinExample.ino (3.5 KB)

If you want a bit more control over the NTP synchronization, which you may, given you are doing sleeps, you may want to look at the ezTime library. It is in the Arduino library manager.
It will give you the ability to control the NTP synchronization where as the code that comes with the core library just runs in the background.

--- bill

BosseB:
Hi Juraj,
I guess you mean the thread Function to retrieve current time on ESP8266 which I started back in May 2018?

no. your first post ESP8266 as a serial port server, how to save parameters? - Microcontrollers - Arduino Forum

and the example in my comment

gfvalvo:
If that's true, won't setup() run again causing another NTP time retrieval?

Exactly! Everything runs from the start so there is no way to know anything except by storing data into EEPROM between executions. The expected execution time each time it wakes up is about 2-3 seconds until it goes to sleep again.
But the sleep interval seems impossible to set accurately unless one goes into the hooplas I am figuring out.

This is why I need NTP in the first place. The plan is:
I save the millis() value and the requested sleep time (or wakeup time) into EEPROM just before going to sleep.
On wakeup after my code gains control I can check when the restart happened by subtracting millis()/1000 from the current time once NTP has synced.
Then I can read back from EEPROM when the restart should have happened and I have the deviation by subtracting the two. Like this:

//wait until NTP has synced (how does one know?)
time_t starttime;
starttime = now() - millis()/1000;
time_t targetstarttime;
EEPROM.get(300,targetstarttime); //address 300 for the stored value
unsigned long timediff;
timediff = starttime - targetstarttime;

Of course, now that I know the diff I can run a delay if the start came up short, but not if I am already late!
Or I can use this to adjust the time correction value so the next sleep will be more accurate.
As you can see I need to wait until NTP has synced successfully with a server, how can I check this in my code?
I have seen that sometimes the NTP reuslts in a failure to sync even though it mostly succeeds...

But the sleep interval seems impossible to set accurately

If you are tying to accurately time a sleep period, you are best off using a DS3231 or other RTC with an alarm interrupt to wake the system.

How long is your sleep interval, and how accurately does it need to be timed?

In the avr world, people have run into similar problems with the accuracy of the watchdog timer oscillator.

This is why I need NTP in the first place. The plan is:
I save the millis() value and the requested sleep time (or wakeup time) into EEPROM just before going to sleep.
On wakeup after my code gains control I can check when the restart happened by subtracting millis()/1000 from the current time once NTP has synced.

I think you will need to save a time_t real time value when you go to sleep and wake up "early" in order to perform the next activities at the precise time you want. Subtracting the last millis() value from the current time, does not give you a meaningful time interval.

cattledog:
If you are tying to accurately time a sleep period, you are best off using a DS3231 or other RTC with an alarm interrupt to wake the system.

I cannot do that because it entails using another device in the design and a way to interface it to the ESP-07. All pins are already used up.

cattledog:
How long is your sleep interval, and how accurately does it need to be timed?

The sleep time would probably in the end be 1 or 2 hours, but while testing I am using 10 minutes. It can be configured in increments of 1 minute in my config code (stored in EEPROM).
What I want is for the readings to be taken at the same second of the set minute.

cattledog:
In the avr world, people have run into similar problems with the accuracy of the watchdog timer oscillator.

I think you will need to save a time_t real time value when you go to sleep and wake up "early" in order to perform the next activities at the precise time you want. Subtracting the last millis() value from the current time, does not give you a meaningful time interval.

The ESP-07 takes different time to get up and running to the point in the sketch where the measurements are initiated. It seems to be because of varying time to effect the network things, connect to the WiFi router, retrieve NTP time etc. It is not the same from execution to execution. I am using the debug serial port to write out the millis value at certain key points and they are different from run to run.

Examples:
setup() is reached in 191-195 ms from start
The start of measurement is reached in 420-743 ms with one outlier case taking 1420 ms

In order to make sure a synced measurement is done I will have to sleep a shorter time by a few seconds and then include a delay at the start of measurement which depends on the time spent sleeping as measured by NTP time saved in EEPROM.

One complication is that I have found that the NTP library sometimes (1 out of 10) reports a sync success at time 2036-02-07 07:28:16. When this happens the time library is happy with that time and it starts counting up from that. It is always the exact same time reported when syncing erroneously as far as I have seen.
I don't know how I can fix this, though. Maybe by skipping the timing adjust delay when it is detected

if (year(now()) == 2036) delayreading(0);
else delayreading(calculated value)

I am starting the NTP service as follows:

if (wifiFirstConnected) {  //One time execution following WiFi connect
  wifiFirstConnected = false;
  NTP.begin ("pool.ntp.org", 1, true, 0);
  NTP.setInterval (60, 86400);
}

And I log the success of the time sync in the event function:

void processSyncEvent (NTPSyncEvent_t ntpEvt)
{
    SerialDebug.print ("Time Sync event");
    switch (ntpEvt)
    {
        case timeSyncd:
            SerialDebug.print (": Time retrieved successfully: ");
            SerialDebug.println (NTP.getTimeDateString(now()));
            ntpTimeSynced = true;
            break;

        case noResponse:
            SerialDebug.println (" error: NTP server not reachable");
            break;

        case invalidAddress:
            SerialDebug.println (" error: Invalid NTP server address");
            break;
    }
}

Of course I have to minimize the "early" time to 2 seconds or so otherwise I will adversely affect the duration of the battery. So I need to be able to hit the right time with some accuracy anyway....
The system runs for about 2 seconds each time it wakes up now.
Back on this tomorrow night...

BosseB:
Of course I have to minimize the "early" time to 2 seconds or so otherwise I will adversely affect the duration of the battery. So I need to be able to hit the right time with some accuracy anyway....
The system runs for about 2 seconds each time it wakes up now.
Back on this tomorrow night...

If you are really power limited, then you may want to consider a combination of an RTC and NTP.
If you use a DS3231, it will be very accurate and you can use it for your time stamps with no need to power up the radio and connect to a network which uses quite a bit of power and takes a bit of time.

Then, to eliminate the RTC drift (which is very minimal with a DS3231)
occasionally, power up the radio and use NTP to set the time in the RTC.
You should not need to resync a DS3231 with NTP more than once a day and should be ok with several days between syncs.

You should also be able to use the Alarm capabilities of the RTC to assert the INT pin which you can use to wake up the processor at a more accurate time so you don't have to wait around for your desired specific time.

--- bill

I see the way an RTC would be able to help in keeping time, however I still need the WiFi to be connected because the measurement result will be transmitted to my webserver's database over the Internet.
And I don't want to add components to my project now. Almost no remaining space on the board.

The bulk of the time now spent in the running state is handling the connection to WiFi and a sensor power-up delay I have randomly set to 1 second. In total the active phase is some 2-2.5 seconds. It gives me about 9 months of operation from 3 AA batteries if measuring once each hour.
I have not checked what the minimum power on of the sensors before measuring can be, I can probably shave off 500 ms or so.
I was thinking that I could use the power-on delay already there by setting the delay time to compensate for the inaccurate deep-sleep time. But I cannot think of a system that would not cause the device to be on typically for much longer times than now and therefore reducing battery life. After all if using a variable delay I have to shorten the sleep time to also handle the worst case difference in sleep clock and the varying startup time when connecting WiFi. If sleep was too long I cannot delay for negative time to compensate...
Probably a better idea would still be to detect how far off the sleep time was in order to calculate the next sleep time.
I.e. automatically adjusting the sleep time correction value I have put in the configuration already.
I could also start the measurements on an even second transition by waiting for now() to change to the next second.
But it seems like overkill, after all what I want is for the measurements to be approximately say every hour and not drift much. A couple of seconds is OK for practical purposes.
Too late in the night now, thinking in circles....

I have set ut a test system tha deep sleeps for 10 minutes and runs a measure cycle.
Here I have debugprinted the NTP sync event showing both the millis() time and the NTP now() time.
Turns out to look OK like this:

Time Sync event: Time retrieved at Millis= 542: 2019-02-08 00:15:00

but now and then there is a message like this:

Time Sync event: Time retrieved at Millis= 723: 2036-02-07 07:28:16

Notice here that unlike what is printed for a normal sync when the time varies it always prints the same timestamp

Since the time_t variable can hold time from 1970 up to 2106 it could not be a top-of-the-range value erroneously recorded. But if NTP counts from 1900 then the end year comes out as 2036.
Why is it always the same bogus datetime when it fails? It should not trigger the success event at all in this case, right?
And what is the time_t of this date (timezone = 1)?

Time Sync event: Time retrieved at Millis= 723: 2036-02-07 07:28:16

Yes, that time date is the 2^32 max date plus one hour.

The result must come from receiving four bytes of 0xFF in the packet buffer passed over to time_t NTPClient::decodeNtpMessage (char *messageBuffer)

Can you please post the code you used to report this time stamp?

I don't know how the values get there, but you can certainly modify the library to test for them and reject the time stamp and get another one.

I didn't read all the of the thread, but esp8266 has built-in RTC. it runs in deep-sleep too (it is used for wake-up). It should keep the time if set.