ESP8266 --Internet Weather Datalogger and Dynamic Web Server

Found the best code, I think: https://github.com/JChristensen/Timezone it is fully customizable. But since it uses TimeLib there is a conflict

dayOfWeek is used in TimeLib and RTClib but is a define in TimeLib

#define dayOfWeek(_time_) ((( _time_ / SECS_PER_DAY + 4) % DAYS_PER_WEEK)+1) // 1 = Sunday

and a function in RTClib

uint8_t dayOfWeek() const;

I added this in the opened issue in TimeLib library but I am thinking to modify one of the libs and put the modified library in the same folder of the main sketch because dayOfWeek is used often and if I change the main library it will break others sketch. The main candidate is RTClib since it has only two files, TimeLib instead has many files.

This issue is well known since years but never resolved.

Edit: I am wrong because I used an old RTClib, the new one uses dayOfTheWeek

Software RTC seems to work well!

Main changes
Hardware RTC removed, it uses now a software RTC that is synchronized every 15 minutes to an NTP server.
RTC is in UTC but Display Clock is automatically adjusted for TimeZone and Daylight Saving Time.

Minor changes

  • Sealevel pressure is now the standard 1013.25, was 1032.4
  • You don’t need anymore to modify the Adafruit BME280 library to use a different I2C address, it uses a #define
  • int year, month, date, hour, minute, second were changed to int year1, month1, date1, hour1, minute1, second1 because were in conflict with TimeLib library.

Other possible improvements

  • Check if it is possible to use a cheap ESP-01 module.
  • Move all user defined parts of code in a separate config.h file (if it is possible!).
  • Recalculate .010 to .020 inches of Mecury for file pressure difference in mBar.

Some test required since i wrote it this morning.

Observations_SPIFFS_20170704.zip (16.3 KB)

@ zoomx

Testing your 08:54 am modifications.

NTP timestamp is wrong in “LOG.TXT” on Sketch startup, if NTP packet is dropped, or has not been received yet. Interval that “ThingSpeak.com” uses is affected in that with no fifteen minute interval data is not sent. No logtoSD function logging occurs when no fifthteen minute interval occurs.

This why I use a real real time clock. RTC are readily available online at low cost.

Real Time Clock is needed.

William

You're right! Need to improve NTP code or drop it. I have to test much more!

Now my brain is online again! I forgot that RTClib can work with an hardware RTC just changing this RTC_Millis Clock; into this RTC_DS3231 Clock;

But there is a difference. To have Timezone and DST working, RTC must be in UTC.

I found the NTP client problem here

sendNTPpacket(timeServerIP); // send an NTP packet to a time server  
// wait to see if a reply is available
  delay(1000);

  int cb = udp.parsePacket();

I need to change the delay into a code that wait for the response with a timeout.

Wemos D1 R2 is back 100%, online now. Attaching Sketch that the Wemos D1 R2 server is currently running.

zoomx is doing some modifications; at this time, I have not added his modifications. Top priority, on project is for downloads to be uninterrupted by events such as logging. Not so much concerned about missing a request to be parsed. Response time is quick and the client closes when a request finishes.

Try a download, think you will be amazed at the speed, compared to before the “readFile” function that martinayotte of ESP8266 Community Forum created.

June 29, 2017, I submitted this project to “Create Arduino Project Hub.” for review and publication.

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

Now featuring four graphs from “ThingSpeak.com:” Graphed Weather Observations Viewable on one web page; trace along the graphed line with your mouse for date/time stamp and value of graphed point.

William

Observations_.SPIFFS.ino.zip (9.62 KB)

Attached there is a test sketch to evaluate the time taken to get a response from an NTP server. I get very short times using the USA NTP pool time.nist.gov

Using the server without hardware RTC is simpler but I think that I can add another define since the switch between hardware and software RTC is very simple, only one line.

It seems that the sketch fit on a cheap ESP-01 module (leaving 64k for SPIFFS) but I have to test it, I have only compiled.

SoftRtcAndNTPusingTimezoneTimeout.zip (2.77 KB)

@zoomx

I tested your NTP code[ attached the result.

Do I understand correctly that you have to continuously hit the time server to get the fifteen minute interval? IMHO, that is a high demand on a time server.

Loop function is only ran, if on the fifteen minute interval; otherwise, Listen function is listening for a request from a client.

Planning to stay with a Real Time Clock for fifteen minute interval, new day, and day of week; to me, it eliminates high demand on a time server.

William

zoomx_NTP_test_results.txt (1.07 KB)

The time request is done only one time every 15 minutes in your sketch modified by me, it can be changed, for example around two AM when in EU there is the daylight time switch.

Instead the test show that you never get a response from the time server. Maybe UDP traffic is firewalled? Test ask for a response and, if it don't get a response within one second make another request. This is repeated 4 times. You get continuous request only because this is a test.

@ zoomx

Re-ran test; Gateway was set to only pass TCP, my apologies. Attached find test run.

Willing to try your code and run it for a couple of days. Are you okay with me doing this?

Using a NTP server is a new experience for me; will need to get up to speed. Thank you.

Attached latest code changes to project as of 7/17/2017.

William

SoftRtcAndNTPusingTimezoneTimeout.txt (2.05 KB)

Observations_SPIFFS.zip (10 KB)

@ zoomx

How often can 0 seconds be detected by NTP server? I have been observing Serial output of your “SoftRtcAndNTPusingTimezoneTimeout.ino” and do not see 0 seconds being detected. If there is no repeatability of 0 seconds; what happens is multiple sets of data gets logged and gets sent to “ThingSpeak.com” until the next minute occurs. Solution?

Your code:

//Write Data at 15 minute interval  
     if ((minute) % 15 == 0){

This code had to be changed:

//Write Data at 15 minute interval  
     if (((minute) % 15 == 0) && ((second) == 0)){

William

I don't check for seconds because there is the possibility that at second zero the server is busy. If it is not busy it will send data to thinghspeak at seconds zero. If it is busy, seconds will be not zero but, for example 20. If I check for senconds == 0 and in this second the server is busy, the data will not be sended to thingspeak because when the server will be free the seconds will be not zero. So when

((minute1) % 15 == 0)

data is sended to thingspeak.

Then there is a flag, AcquisitionDone, that became true after sending the data to thingspeak. The next time, if minute is still ==0, this

((minute1) % 15 == 0)

is true but this

(AcquisitionDone == false)

is false because now AcquisitionDone is true. So no data is sended to thingspeak again. When

((minute1) % 15 == 0)

became false, AcquisitionDone became false too so the next time that

((minute1) % 15 == 0)

will be true, AcquisitionDone is false and data is sended to thingspeak. I wroted this some months ago for an SD datalogger because I have some of them and I want to have all data synchronized. This send data only every 15 minutes and synchronize RTC with NTP only every 15 minutes too

@ zoomx

Thank you.

Testing your "Observations_SPIFFS_20170704.ino" I have been unable to open web pages; /Weather, /SdBrowse, or /Graphs. Have you experience this in your testing? Is there a conflicting library?

"Observations_SPIFFS.ino" uses ESP8266Wifi library for both client and server, without your modifications; /Weather, /SdBrowse, and /Graphs web pages open. I even tried with just listen function in loop; still your version does not open web pages.

Do you have a later version for testing?

William

You have to add the FAVICON.ICO file into SPIFFS. Without this file you can't reach any other page. You can see this in the serial debug. Just tested, I can see all pages.

I have to add the last update that you wrote.

@ zoomx

All is well… added delay(500); before listen function call in loop; may need tweaking.

Unzipped your “Observations_SPIFFS_20170704.zip” file; made local port, 123.

All web pages opening as you stated. Thank you zoomx.

William

Testing modded “Observations_SPIFFS_20170704.zip” with my latest update; Serial output attached, check out between 7/19/2017 @ 23:00 and 7/19/2017 @ 23:15. packet was received; however, “Seconds since Jan 1 1900 = 0” Should this condition be tested? Causes timestamp to be wrong for log file.

William

Serial output --modded file.txt (3.14 KB)

Message received, I have to check what happens if you don't get response from NTP or get an invalid response to avoid to set the RTC to a wrong time. Looking at code, if no packet is received there is no update time. I have to add another check to the packet length, it must be 48. In Wikipedia https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation I read that UDP packet has a checksum so I believe that the packet received should be correct. If packet is not correct it should be discarded and it's like not receiving any packet. Maybe it can happen that the server is wrong or something happens. Long time ago it happened that GPS went wrong and an automatic GPS data collection didn't start because of this strange error. If you vave frequent NTP errors it's better that you mantain the DS3231 RTC, as you can see it's only a line different.

Anyway Thingspeak uses it's time server and you can download data from it. In my opinion Thingspeak is great since you can build your own server using a PC or a little RaspberryPI so if they change idea (pachube, cosmic now Xively changed idea!) you can have your server. Roberto

Thank you zoomx of Arduino.cc Forum for implementing NTP with Timezone for the project! Tested for two days; time intervals , logging and ThingSpeak are close to working correctly. Perhaps some small tweaking remains. Well done zoomx.

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Thanks but I made only a small contriubution! I thought to use NTP because I am lazy and I prefer a clock that regulate itself. Two time a year I have to regulate many clock when there is the switch to the Daylight Saving Time!

Two days of testing NTP with Timezones, I am convinced it is not redy for prime time. Not sure if it is a library or NTP, or a hardware issue. Server randomly losses track of NTP timing interval. I am reverting back to using a DS3231 RTC.

Wondering if it could be the ESP8266 going into a sleep mode and not getting NTP packets.

Returned to Real Time Clock version; attached file is current running the Wemos D1 server.

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Observations_SPIFFS.zip (10 KB)