NTP --ESP8266 Time-Synced, Data Logging and Web Server

04/24/2018

“NTP_Time_synced_Web_Interface.ino" has obtained faster performance using "ESP8266 sys/time.h" and "ESP8266 time.h" libraries suggested by schufti of the ESP8266 Forum. Schufti provided example code for using the two libraries. Removed some code related to earlier libraries and three functions related to NTP, UDP, and and the old Time keeping code. Everything is complete: time related functions, web pages, file listing, file downloading, and graphing! ESP8266 server website is averaging less than one second response times from client connect, serving the webpage, and client closes! Response time can be monitored with Serial Monitor from time stamping.

More than a Data Logger; also allows fast, remote downloading of files, stored onboard in the SPIFFS memory. Files are listed and selectable for download. Both ESP8266, local server and hosted, web server receives uploaded data from the same sketch. Commented out are two features; a LCD display function and an experimental, audio beep function to alert for a predetermined difference in a 15-minute interval of barometric pressure. Have read of past references of serve weather associated with a sudden drop in barometric pressure.

RobotDyn WiFi D1 board that is online has been used for development of project coding; therefore, there may be gaps in the data logs! The 15-minute interval has been "spot on" every 15 minutes; awesome, since project has no RTC module! All time related requirements are provided by NTP time server; even has a second time server coded!

ESP8266 hosted, website and Domain, hosted website One sketch dynamically, updating data to both websites.

Restrictions of free, domain hosting service is why there is no file browser or file downloading on the domain hosted website.. Graphing of four, weather parameters is available on both websites.

Thank you, “Arduino,cc Forum,” “ESP8266 Community Forum,” and “Adafruit Forum” for all the help received with this Open-Source project!

05/05/2018 updated with current project file.

William

Serial Monitor --no Client IP.txt (8.54 KB)

yourOwnFilename.zip (1.09 KB)

NTP_Time_synced_Web_Interface.zip (11.1 KB)

05/07/2018

Video discussing "Arduino IDE" multiple tabbed compiling.

Technique allows better management of complex, long project code with less scrolling when re-writing , adding, or changing code.

Unzip attached file to a folder; in this folder double click "NTP_Time-synced_Web_Interface.ino" to open the "Arduino IDE" with tabs. First tab will be always be includes, declarations, globals, setup, and loop. Rest of the tabs are functions.

William

NTP_Time-synced_Web_Interface.zip (15.6 KB)

Current Features "NTP_Time-Synced_Web_Interface.ino".

1. NTP Server; used for 15 minute time interval, date-time stamping, and dayofweek (every 6th
   day, "LOG.TXT" file gets renamed to keep file size manageable. Every Saturday "LOG.TXT" gets
   renamed in the format "logxxyy" xx being the month and yy being the date; a new log.txt is
   created after every file renaming.

2. Dynamic web page of current observations for Last update time and date, humidity, dew
   point, temperature, heat index, barometric pressure (both inches of Mercury and millibars.)

3. Server files are listed as web links; clicking link prompts for "Open with/Save as."
   "FAVICON.ICO", and "ACCESS.TXT" are listed; however, they are for internal use and cannot
   "Opened with/Save as," result of clicking link produces "404 Page not found."

4. LOG.TXT file is appended every 15 minutes with the latest update; storing data from Dynamic
   web page.

5. ACCESS.TXT restricted file that logs client IP address.

6. DIFFER.TXT stores the difference in Barometric Pressure for the last fifteen minutes. Only
   a difference of equal to or greater than .020 inches of Mercury are logged with difference,
   date and time.

7. URL file names other than ones defined in the Sketch produce "404 Page not found." Methods
   other than "GET," produce "405" message and exits current request.

8. Audible alert from Piezo electric buzzer when there is Barometric Pressure difference of
   .020 inches of Mercury. I am interested in sudden drop of Barometric Pressure in a 15 minute
   interval. Serve weather more likely with a sudden drop. Difference of .020 inches of Mercury
   point is set for my observations to log and sound audible alert; not based on any known value
   to be associated with serve weather.

9. Two-line LCD Display of Barometric Pressure, in both inches of Mercury and millibars.

10. Tempature, Humidity, Barometric Pressure, and Dew Point have four embedded "ThinkSpeak.com"
    graphs on one web page. Graphs are created from Iframes provided by "ThingSpeak.com"

11. ThingSpeak.com is used for graphing four weather parameters. iFrames are obtained from
    ThingSpeak..com.

12. Free, "000webhost powered by HOSTINGER" is used for second website.

Two websites,one sketch: "NTP_Time-Synced_Web_Interface.ino."

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

Note this is a project is in development; maybe offline or log files may be affected.
Server is online 24/7; except during periods of testing.

Server is a "RobotDyn WiFi D1 R2" with 32 MiB Flash Memory. Development Board, purchased from
"RobotDyn.com" and GY-BME280 breakout board, purchased from "Ebay.com" ; both are required for
project. Sensor is located indoors, currently.

Developement board is Arduino friendly; can be programed using the Arduino IDE.
Under $15.00 with shipping, is the project cost for Development Board --$4.29 and one,
GY-BME280 breakout board, sensor for Barometric Pressure, Humidity, and Temperature --cost $4.77.

Short, Serial Monitor Output:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.05.22 00:27:35 =~=~=~=~=~=~=~

Client connected: Tue , 05/22/2018 , 00:25:05 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /Weather
Client closed: Tue , 05/22/2018 , 00:25:06 EDT

Client connected: Tue , 05/22/2018 , 00:25:10 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /SdBrowse
Client closed: Tue , 05/22/2018 , 00:25:11 EDT

Client connected: Tue , 05/22/2018 , 00:25:14 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /LOG.TXT
File: /LOG.TXT
Client closed: Tue , 05/22/2018 , 00:25:25 EDT

Client connected: Tue , 05/22/2018 , 00:25:57 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /README.TXT
File: /README.TXT
Client closed: Tue , 05/22/2018 , 00:25:58 EDT

Client connected: Tue , 05/22/2018 , 00:26:27 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /Graphs
Client closed: Tue , 05/22/2018 , 00:26:28 EDT

Client connected: Tue , 05/22/2018 , 00:26:38 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /Graphs
Client closed: Tue , 05/22/2018 , 00:26:39 EDT

Client connected: Tue , 05/22/2018 , 00:26:42 EDT
Client IP: 10.0.0.146
Processing request
Action: GET
Path: /Graphs
Client closed: Tue , 05/22/2018 , 00:26:43 EDT

Attached is the current tabbed, Arduino IDE version of the project

NTP_Time-synced_Web_Interface.zip (15.5 KB)

Due to circumstances beyond my control; project has been offline since 05/29/2018. Online operation was restored 06/06/2018 at 01:30 EDT. ISP changed from Gateway -based firmware to Gateway cloud -based firmware; there were issues with port forward not working. Gateway was swapped out for a new one in the troubleshoot process and a new public IP was attached nullifying previous links to the ESP8266 Server web site.

I apologize for any inconvenience experienced by this change.

Active project websites:

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Experienced sensor failure; awaiting replacement sensor. Both websites are affected; updated data is invalid!

Affected websites:

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

Update 7/9/2018

Added WiFiStart call to restart WiFi if there is a WiFi disconnect; to listen function. New function wifiStart is used in setup and listen functions.

Code added to log WiFi reconnects and date/time stamp each occurrence; to listen function:

if(WiFi.status() != WL_CONNECTED)
  {
      wifiStart();

      delay(1000 * 10);   //wait 10 seconds before writing 

     //Open a "WIFI.TXT" for appended writing.   Client access ip address logged.
     File logFile = SPIFFS.open("/WIFI.TXT", "a");

     if (!logFile)
     {
          Serial.println("File failed to open");
     }
     else
     {
          logFile.print("Reconnected WiFi:  ");
          logFile.println(dtStamp);
     }

  
  }

wifiStart function:

void wifiStart()
{  
  
  WiFi.mode(WIFI_STA);

  Serial.println("");
  Serial.print("MAC: ");
  Serial.println(WiFi.macAddress());

  // We start by connecting to a WiFi network
  Serial.print("Connecting to ");
  Serial.println(ssid);
  

  //setting the addresses   
  IPAddress ip(10,0,0,13);
  IPAddress gateway(10,0,0,1);
  IPAddress subnet(255,255,255,0); 
  IPAddress dns(75,75,75,75);

  WiFi.begin(ssid, password);

  WiFi.config(ip, gateway, subnet, dns);
  
  while (WiFi.status() != WL_CONNECTED) 
  {
    delay(100);
    Serial.print("."); 
  }

  WiFi.waitForConnectResult();

  Serial.printf("Connection result: %d\n", WiFi.waitForConnectResult());

}

Over rode "Auto Modem Sleep" default sleep mode; power no issue, using AC to DC adapter.

wifi_set_sleep_type(NONE_SLEEP_T);

Default "Auto Modem Sleep" I believe is too slow "waking," causing Browser request to occasionally timeout. Continuing to observe this solution.

William

NTP_Time-synced_Web_Interface.zip (16.9 KB)

July 15, 2018

Moved block of code responsible for every 15 minute data updates, logging, sending data to ThingSpeak and the Hosted, domain website moved from loop to inside of listen function. Listen function has priority except for 10 seconds at 14 min 50 sec., 30 min. 50 sec., 45 min. 50 sec., and 59 min 50 sec.; then the moved code block has priority. Result of this move has proved to be better responses to Client requests and improved reliability!

WiFi Client/Server Based --ESP8266

Domain Hosted, WiFi Client.Server Based --ESP8266

William

NTP_Time-synced_Web_Interface.zip (16.9 KB)

WiFi Client/Server Based --ESP8266

There are two web sites; one hosted on the ESP8266, and one domain, hosted --both are driven by one sketch!

Domain Hosted, WiFi Client.Server Based --ESP8266

User's countries.JPG713×561 39.7 KB

08/08/2018 Current version of sketch is attached.

Robotdyn WiFi D1 Project.jpg800×533 116 KB

RobotDyn WiFi D1 Development Board and BME280 Sensor mounted on SB400 proto-typing board. Sensor mounted on 6 pin, female header. Making for easy replacement and/or testing of BME280 sensors.

User's countries.JPG713×561 39.7 KB

NTP_Time-synced_Web_Interface.zip (17 KB)

Robotdyn WiFi D1 Project.jpg800×533 116 KB

Hi dear you project is very beautiful and stunning, I have one suggestion if you add GPS module like neo-6m or similar to get time from a satellite. thank you

@ haroon552

Thank you for your feedback and suggestion. Presently, the additional benefits of satellite time are not required. I have noted your suggestion. Thank you haroon552.

GPs to add Coordinates and time and how can we add the tipping bucket rain gauge?

haroon552

Glad you have an interest in additional features; unfortunately, I do not have time to design, code this additional feature now.

May I suggest joining a makerspace to pursue your feature request. You can find a makerspace (world-wide) in the directory at: https://spaces.makerspace.com/

This article might get you started on adding a tipping bucket rain gauge: https://www.instructables.com/id/NodeMCU-Hall-Effect-Rain-Gauge/

Here is a retail, wireless device that you might modify and decode the 433 Mhz protocal; however, modification and decoding is beyond the scope of this post:

AcuRite-00899-Wireless-Self-Emptying-Collector, user manual: https://images-na.ssl-images-amazon.com/images/I/A1uOyWTqMLL.pdf

SwitchDoc weather related project, Arduino/Raspberry PI based (with open source software):

Recommend reading reviews for the above item from SwitchDoc to know additional requirements to get up and running with this item.

William

I appreciate your help in this regard thank you, sir, I am building a small project, using your project to amend and add some rain and GPS features when it complete I will show you, and if I need any help I will ask you,

haroon552

Found this code for tipping rain gauge:

UNSWmir's repository

Rain gauge using NodMCU with hall effect sensor

Includes instructables tutorial and 3D printer files.

William

I already added the function of rain that is working fine that is saving the rain, but the problem i am faceing while adding tinygps++ library, code is crashing again n again and even not getting time n date , i am using ublox neo6m gps module on software serial D6-D7 nodemcu pins.

THREE FILES ARE ATTACHED

1010181436adding_Rain_TempNo_HumidityNo_NTP_Time-synced_Web_Int.zip (20.5 KB)

nodemcu_gps_neo_6m_test_mod_ready.zip (1.77 KB)

GPS_Time_NOT_WORKING1010181436adding_Rain_TempNo_HumidityNo.zip (20.4 KB)

If you are only using GPS time and date remove the following code:

#include <sys/time.h>                   // struct timeval
#include <time.h>                       // time() ctime()

I can not say this is a fix; just a suggestion. Reinstalling TinyGPSPlus library may fix the problem.

Load your sketch into Arduino IDE. Select "Sketch" tab, go down and select "Include library," slide mouse over to next column and select "Add ZIP library." Go to location of zipped library file and select the file. You can remove the line it places at the top of your sketch; since it is already in the sketch.

Once the library is installed; this is what I see using Notepad++ for Arduino IDE code editing:

Note when I looked at your code, "date" was not red, font color. This could have been from my library setup; thought it was worth mentioning.

Noticed you are adapting an older version of "NTP_Time-Synced_Web_Interface.ino." Attached is the current version of the project that is driving two different web sites. See previous post for web site links.

Another suggestion; look at the "TinyGPSPlus" library examples, comparing to see if anything could have been missed when adapting to my project.

Since I do not have a GPS module to test with; I can only make suggestions.

William

NTP_Time-synced_Web_Interface.zip (66.1 KB)

haroon552

Which board are you using for your project? My project uses an ESP8266 12E. Is this code for a NodeMCU 12E; if so then this applies...

Looking at your "Not working" code again; I see in setup you are using "wire.begin()" this I believe should be "wire.begin(6,7);" as these are the pins you are using your softwareserial library. One other note; are you using the "espsoftserial" library?.

Reference as to why to use "espsoftserial library.".

According to this reference you should use a lower speed baud rate; reference author recommends 9600 baud.

Additionally, read this post about esp8266 system reserved pins.

NodeMCU pinouts unavailable pins Also pinouts may be board specific; check vendors support page for pinouts.

William

hi,
I am using Esp8266 12e board fo this project . i reinstalled the library . still same. actually it reboots again and again just it connect to wifi and got ip after it reboots . could you please solve my problem

haroon,

Please download the library I linked to in the modified source code. The arduino uno/mega version is reported not to work with the ESP8266. Please download, compile and try attached code before making any other changes, Do install the "espsoftwareserial" library to the Arduino IDE.

There are changes; by line numbers...

50-51 Commented out two unneeded libraries from includes.
70-75 Removed SSID and passwords
181-183 Removed ThingSpeak channelnumber and apikey
188-189 Defined GPSBaud; it was undefined
196-201 Assigned RX and Tx pins for SoftwareSerial

I have no NEO6M GPS to test this code; please test and let me know result, thank you.

YouTube video:

ESP8266 Nodemcu & Ublox GPS & Oled display location tinygps tutorial

William

Try__GPS_Time_NOT_WORKING1010181436.zip (20 KB)

If you give me address I can gift you the neo GPS module, please

I modified the code, It is working but the problem is server becomes slow and I thing SoftwareSerial buffer is getting full, and could you please add delay for GPS, it should wait until date and time obtained when logger start.

and one more question how can I send the saved log.text file to FTP server that is saving inside this sketch. mean for the whole day I have no wifi when I connect wifi it should connect automatically to FTP server and upload file automatically there is anything like this?