ESP8266 --Internet Weather Datalogger and Dynamic Web Server

Attached there is a test sketch to evaluate the time taken to get a response from an NTP server. I get very short times using the USA NTP pool time.nist.gov

Using the server without hardware RTC is simpler but I think that I can add another define since the switch between hardware and software RTC is very simple, only one line.

It seems that the sketch fit on a cheap ESP-01 module (leaving 64k for SPIFFS) but I have to test it, I have only compiled.

SoftRtcAndNTPusingTimezoneTimeout.zip (2.77 KB)

@zoomx

I tested your NTP code[ attached the result.

Do I understand correctly that you have to continuously hit the time server to get the fifteen minute interval? IMHO, that is a high demand on a time server.

Loop function is only ran, if on the fifteen minute interval; otherwise, Listen function is listening for a request from a client.

Planning to stay with a Real Time Clock for fifteen minute interval, new day, and day of week; to me, it eliminates high demand on a time server.

William

zoomx_NTP_test_results.txt (1.07 KB)

The time request is done only one time every 15 minutes in your sketch modified by me, it can be changed, for example around two AM when in EU there is the daylight time switch.

Instead the test show that you never get a response from the time server. Maybe UDP traffic is firewalled?
Test ask for a response and, if it don't get a response within one second make another request. This is repeated 4 times. You get continuous request only because this is a test.

@ zoomx

Re-ran test; Gateway was set to only pass TCP, my apologies. Attached find test run.

Willing to try your code and run it for a couple of days. Are you okay with me doing this?

Using a NTP server is a new experience for me; will need to get up to speed. Thank you.

Attached latest code changes to project as of 7/17/2017.

William

SoftRtcAndNTPusingTimezoneTimeout.txt (2.05 KB)

Observations_SPIFFS.zip (10 KB)

@ zoomx

How often can 0 seconds be detected by NTP server? I have been observing Serial output of your "SoftRtcAndNTPusingTimezoneTimeout.ino" and do not see 0 seconds being detected. If there is no repeatability of 0 seconds; what happens is multiple sets of data gets logged and gets sent to "ThingSpeak.com" until the next minute occurs. Solution?

Your code:

//Write Data at 15 minute interval  
     if ((minute) % 15 == 0){

This code had to be changed:

//Write Data at 15 minute interval  
     if (((minute) % 15 == 0) && ((second) == 0)){

William

I don't check for seconds because there is the possibility that at second zero the server is busy. If it is not busy it will send data to thinghspeak at seconds zero. If it is busy, seconds will be not zero but, for example 20. If I check for senconds == 0 and in this second the server is busy, the data will not be sended to thingspeak because when the server will be free the seconds will be not zero.
So when

((minute1) % 15 == 0)

data is sended to thingspeak.

Then there is a flag, AcquisitionDone, that became true after sending the data to thingspeak.
The next time, if minute is still ==0, this

((minute1) % 15 == 0)

is true but this

(AcquisitionDone == false)

is false because now AcquisitionDone is true.
So no data is sended to thingspeak again.
When

((minute1) % 15 == 0)

became false, AcquisitionDone became false too so the next time that

((minute1) % 15 == 0)

will be true, AcquisitionDone is false and data is sended to thingspeak.
I wroted this some months ago for an SD datalogger because I have some of them and I want to have all data synchronized.
This send data only every 15 minutes and synchronize RTC with NTP only every 15 minutes too

@ zoomx

Thank you.

Testing your "Observations_SPIFFS_20170704.ino" I have been unable to open web pages; /Weather, /SdBrowse, or /Graphs. Have you experience this in your testing? Is there a conflicting library?

"Observations_SPIFFS.ino" uses ESP8266Wifi library for both client and server, without your modifications; /Weather, /SdBrowse, and /Graphs web pages open. I even tried with just listen function in loop; still your version does not open web pages.

Do you have a later version for testing?

William

You have to add the FAVICON.ICO file into SPIFFS. Without this file you can't reach any other page. You can see this in the serial debug.
Just tested, I can see all pages.

I have to add the last update that you wrote.

@ zoomx

All is well... added delay(500); before listen function call in loop; may need tweaking.

Unzipped your "Observations_SPIFFS_20170704.zip" file; made local port, 123.

All web pages opening as you stated. Thank you zoomx.

William

Testing modded "Observations_SPIFFS_20170704.zip" with my latest update; Serial output attached, check out between 7/19/2017 @ 23:00 and 7/19/2017 @ 23:15. packet was received; however, "Seconds since Jan 1 1900 = 0" Should this condition be tested? Causes timestamp to be wrong for log file.

William

Serial output --modded file.txt (3.14 KB)

Message received, I have to check what happens if you don't get response from NTP or get an invalid response to avoid to set the RTC to a wrong time.
Looking at code, if no packet is received there is no update time. I have to add another check to the packet length, it must be 48.
In Wikipedia

I read that UDP packet has a checksum so I believe that the packet received should be correct. If packet is not correct it should be discarded and it's like not receiving any packet.
Maybe it can happen that the server is wrong or something happens. Long time ago it happened that GPS went wrong and an automatic GPS data collection didn't start because of this strange error.
If you vave frequent NTP errors it's better that you mantain the DS3231 RTC, as you can see it's only a line different.

Anyway Thingspeak uses it's time server and you can download data from it. In my opinion Thingspeak is great since you can build your own server using a PC or a little RaspberryPI so if they change idea (pachube, cosmic now Xively changed idea!) you can have your server.
Roberto

Thank you zoomx of Arduino.cc Forum for implementing NTP with Timezone for the project! Tested for two days; time intervals , logging and ThingSpeak are close to working correctly. Perhaps some small tweaking remains. Well done zoomx.

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Thanks but I made only a small contriubution!
I thought to use NTP because I am lazy and I prefer a clock that regulate itself. Two time a year I have to regulate many clock when there is the switch to the Daylight Saving Time!

Two days of testing NTP with Timezones, I am convinced it is not redy for prime time. Not sure if it is a library or NTP, or a hardware issue. Server randomly losses track of NTP timing interval. I am reverting back to using a DS3231 RTC.

Wondering if it could be the ESP8266 going into a sleep mode and not getting NTP packets.

Returned to Real Time Clock version; attached file is current running the Wemos D1 server.

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Observations_SPIFFS.zip (10 KB)

Testing. Data will be here

Since it is at work I cannot have a public access to the server. I will have it when I will move it to my home.
This is just a test for RTC and logging.

Good morning,

Re-visited zoomx's application of "NTP with Timezones and have a successful working sketch, "NTP_Observations_SPIFFS.ino"

Included in the attachment are the sketch, "LOG.TXT (portion actually logging during testing) and output of the Serial Monitor (during testing.)

Note: Time keeping is now accomplished with software using NTP time server.

William

NTP Observatons_SPPIFFS .zip (15.8 KB)

Server and sensor running "NTP_Observations_SPIFFS.ino" with No Real Time Clock!

Server has been online since 7/29/2017; all time keeping functions are handled by NTP time server.
Thank you "zoomx" for applying NTP and Timezones to project!

Check out thefour graphs from "ThingSpeak.com;" let mouse hover over data point on graph to reveal value and time of data entry:

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

zoomx:
I read that UDP packet has a checksum so I believe that the packet received should be correct. If packet is not correct it should be discarded and it's like not receiving any packet.

I am wrong and this assumption seems false.

In Serial output --modded file.tx

packet received, length=12
Seconds since Jan 1 1900 = 0
Unix time = 2085978496


Data written to log  02/07/2036 , 01:30:00

there was a packet that was long 12 bytes instead of 48.
So packet with lenght different from 48 must be discarded.

I apologize William, you wrote this already.

@ Zoomx

No need to apologize zoomx. Thank you for all your time, coding, and community participation.

Logging of Data now has priority after too many missed 15 -minute intervals.

This version has been tested over three days; much improved, over previous versions. All time keeping is NTP based; no Real Time Clock!

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Serial Output --log priority --8-11-2017.txt (73.6 KB)

Testing --LOG.TXT (47.4 KB)

NTP_Observations_SPIFFS.zip (11.7 KB)

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266
3 Minute Video of project

William

NTP_Observations_SPIFFS.zip (11.7 KB)