Arduino Forum

Community => Exhibition / Gallery => Topic started by: Techno500 on Mar 30, 2017, 08:49 pm

Title: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Mar 30, 2017, 08:49 pm
Ported my Arduino Mega project, "SdBrowse_CC3000_HTTPServer.ino" over to ESP8266EX based, WeMOS D1 R2 Development Board (Arduino compatible, with the exception of some libraries.)

WeMOS D1 R2  (https://www.wemos.cc/product/d1.html)

During the porting of project martinayotte of the ESP8266 Community Forum converted two functions; listFiles and readFile functions.  Along the way on the original projects there was help from Arduino.cc Forum, Adafruit Forum, Arduino Stack Exchange.

Project uses many reusable functions; logtoSD, lcdDisplay, listen, listFiles, readFile, ReadDS3231, getDateTime, getWeatherDate, updateDifference, newDay, and fileStore.

ESP8266   --Internet Weather Datalogger and Dynamic Web Server

March 30, 2017 Current Features of renamed project, "Oberservations_SPIFFS.ino".  SPIFFS is the Flash meory system built-in the WiFi module on weMOS board.

Requires WeMOS D1 R2 Development Board, Real time clock (DS3231) and Barometric Pressure, Humidity/Temperature sensor (BME280).

1. Real Time Clock; used for 15 -minute time interval, date and time stamping and dayofWeek (every 0 day of the DS3231, log.txt file gets renamed to keep file size manageable. Every Saturday (0 day of week) log.txt gets renamed in the format "logxxyy" xx being the month and yy being the date; a new log.txt is created after every file renaming.

2. Dynamic web page of current observations for Last update time and date, Humidity, dew point, temperature, heat index, barometric pressure (both inches of Mercury and millibars.)

3. SPIFFS files are listed as web links; clicking link prompts for "Open with/Save as." "System~1/", "Favicon.ico", and "Access" are listed; however, they are for internal use and cannot "Opened with/Save as," result of clicking link produces "404 Page not found."

4. LOG.TXT file is appended every 15 -minutes with the latest update; storing data from Dynamic web page.

5. ACCESS.TXT restricted file that logs client IP address.

6. DIFFER.TXT stores the difference in Barometric Pressure for the last fifteen minutes. Only a difference of equal to or greater than .020 inches of Mercury are logged with difference, date and time.

7. URL file names other than ones defined in the Sketch produce "404 Page not found."

8. Audible alert from Piezo electric buzzer when there is Barometric Pressure difference of .020 inches of Mercury. I am interested in sudden drop of Barometric Pressure in a 15-minute interval. Serve weather more likely with a sudden drop. Difference of .020 inches of Mercury point is set for my observations to log and sound audible alert; not based on any known value to be associated with serve weather.

9. Two-line LCD Display of Barometric Pressure, in both inches of Mercury and millibars.
Server is a WeMOS D1 R2, Model 2.1.0 Development Board -- purchased on Ebay for ~ $ 9.00 and BME280 breakout board and DS3231 breakout board ~ $ 6.00; both are required for project. Sensor is located indoors, currently.


Experience with the WeMOS has been a very good; downloads are 15 times faster, WiFi connectivity is MUCH improved, reliability near 100% thanks to on-board watch dog timer (which, is enabled by default.), WiFi interference is a non factor --gone.


Temporary project web site (http://tinyurl.com/Observations-weather)


Check out the SPIFFS Files link; select a file and time the download!

Update:  August 2, 2017

New update; RTC DS3231 replaced with NTP Time server and Timezones applied by "zoomx" of Arduino Forum.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Apr 25, 2017, 09:45 pm
Update 4/25/2017

Function newDay was not correctly storing current week of Data.  Made correction and tested; function newDay is correctly detecting Saturday and the time specified, and renaming file based on the month and date and creating a new LOG.TXT file for new week of Data.

Wemos D1 R2 Developement board continues to perform very well; much better reliability than what I experienced with the Adafruit CC3000 Shield.

William 
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 06, 2017, 11:30 pm
Online since March 30, 2017; project has maintained availability and reliability.  WEMOS D1 R2 Developement board continues to perform very well; much better reliability than what I experienced with the Adafruit CC3000 Shield with the Arduino Mega 2560.  File system on the Wemos D1 is built-in to the Wifi module/processor.  There is no SD card to deal with having to be reset.

WEMOS D1 development board has been a very good experience.

Project web page --Weather Data observations (http://tinyurl.com/Observations-weather)


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 16, 2017, 02:18 am
Added sending of "favicon.ico"   updated 06/15/2017 @ 20:00 EST.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 24, 2017, 03:01 am
Wemos D1, Weather Observations project now with embedded Graphing of Temperature, Humidity, Barometric Pressure, and Dew Point.  All previous feature are still included.  Web link is provided to "Thingspeak.com" Channel; which provides the graphing, updated every fifteen minutes..

Wemos D1, Weather Observations project (http://tinyurl.com/Observations-weather)

Some lines will require editing with your information.  Lines requiring editing have been commented in the source code.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 26, 2017, 10:14 am
If you have space in flash you can use the WiFiManager library
https://github.com/tzapu/WiFiManager (https://github.com/tzapu/WiFiManager)
thus avoiding to modify the source just to set the WiFi SSID and password.
In this way the ESP8266 can use more than only one settings.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 26, 2017, 12:11 pm
@ zoomx   Thank you for the tip, appreciate your input.

Project updated; consolidated two web pages for graphs into one page with graphs configured in a two row by two column configuration.

Download, Observations_SPIFFS.ino has commented lines that require your editing to be useable.


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 27, 2017, 11:26 am
Mybe I found a bug. This
Code: [Select]
if ((bme_temp <= 26.66) || (bme_humidity <= 40)) heat_index = bme_temp;
Must be changed in this
Code: [Select]
if ((bme_temp <= 26.66) || (bme_humidity <= 40)) heat_index = T;
because bme_temp is in Celsius and T is in Fahrenheit.
I found it while changing the sketch to use Celsius.


Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 28, 2017, 01:06 pm
Thank you zoomx.  Does simplify and eliminate the conversion to Fahrenheit in the logtoSD and listen functions.  Or do we want to leave it the way it was for readability? 

Project web page.  Now with Graphed Weather Observations (http://tinyurl.com/Observations-weather)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 28, 2017, 01:55 pm
The change suggested is useful only if you use Fahrenheit because when the if condition is met, bme_temp is assigned to heat_index but bme_temp is in Celsius. So you have to use T.

I will rewrite the changes I made using #IFDEF so it will be very simple to change from C to F using only one #define, then I will post here the sketch.

Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 29, 2017, 12:50 am
@ zoomx  Like your coding...

Will be very good to have a single place to globally change from Fahrenheit to Celsius when changing temperature units.   Thank you.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 29, 2017, 10:54 am
Maybe I found another bug
Code: [Select]
log.print((bme_pressure + 27.47), 2);  //Convert hPA to millibars
As far as I know 1mbar = 1hPa so you don't need any conversion.

Edit: found also this one
Code: [Select]
client.println(((bme_pressure) + 27.47), 1);   //Convert hPA to millbars
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 29, 2017, 11:38 am
Ok, attached there is the modified sketch. TEST IT since I ave no ds3231 free.

I leaved the pressure difference still in hg inch because I need to recalculate the limits.

I am changing lib to RTClib. This way I can use a DS3231 or use a soft RTC (if it works?) and use NTP call to set the correct local time.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 29, 2017, 11:56 am
@ zoomx

27.47 is a correction factor. When compared to a reference sensor on another project and nearby Weather U.underground Barometric Pressure reading in millibars,  current sensor was reading low.  Will add comment to about correction factor for my sensor.  Over-sight in not commenting  correction factor.

Function "listen" has same correction factor:

Code: [Select]
client.println(((bme_pressure) + 27.47), 1);   //Convert hPA to millibars

Will place the correction factors in the "getWeatherData" function like I did for "currentPressure," for consistency.


Will make the changes in next update.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 29, 2017, 12:24 pm
Maybe better to use a const float (or a #define) at the beginning of the sketch so one can change it easily.

I have a BME280 and a BMP180 online but they differ of only 1mBar. The BME280 is attached to a nodeMCU and is this one that I will use with your sketch.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 29, 2017, 01:34 pm
@ zoomx

Current BME280 on the project board in a China variant; maybe way out of tolerance.  Have requested a replacement; hopefully better, from another source.  

Tested "Observations_SPI_CF.zip"  Works as you coded; see attached file."

I agree on the tolerance of  the Bosch BME280.  Unknown if China BME280 is the "real deal."

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 29, 2017, 02:06 pm
Great!
I don't have any reference barometer, maybe I can use the value provided by the nearby airport.
My Bosh sensor came from China too.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jun 29, 2017, 03:55 pm
software RTClib  seems to work, I have to add NTP request.

Maybe you can change this
Code: [Select]
 //Write Data at 15 minute interval
    if ((((minute) == 0) ||
         ((minute) == 15) ||
         ((minute) == 30) ||
         ((minute) == 45))
        && ((second) == 0))
    {
      lastUpdate = dtStamp;   //store dtstamp for use on dynamic web page
      getWeatherData();
      updateDifference();  //Get Barometric Pressure difference
      logtoSD();   //Output to SD Card  --Log to SD on 15 minute interval.
      delay(10);  //Be sure there is enough SD write time
      speak();
      //lcdDisplay();      //   LCD 1602 Display function --used for 15 minute update
    }
    else
    {
      listen();  //Listen for web client
    }


with this
Code: [Select]
//Write Data at 15 minute interval  
if ((minute) % 15 == 0) {
    if (AcqDone == false) {
      lastUpdate = dtStamp;   //store dtstamp for use on dynamic web page
      getWeatherData();
      updateDifference();  //Get Barometric Pressure difference
      logtoSD();   //Output to SD Card  --Log to SD on 15 minute interval.
      delay(10);  //Be sure there is enough SD write time
      speak();

      AcqDone = true;
    }
  }
  else
  {
    AcqDone = false;
    listen();  //Listen for web client
  }


and add a
Code: [Select]
bool AcqDone = false;
at the beginning.

This way you can change the interval easily. The other advantage is that maybe if you get  a GET request just few seconds before the acquisition you can skip the sending data to ThingSpeak since seconds will be not zero anymore.
The AcqDone flag is used to avoid to send multiple data during the minute.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 01, 2017, 01:34 pm
Update:  Web site for project is down; maybe a few days.

Web site Down for maintenance (http://tinyurl.com/Observations-weather)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 01, 2017, 08:31 pm
I tested successfully the software RTC with NTP but I am in search of a code to change automatically for DayLight Saving Time, USA and Europe.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 03, 2017, 12:38 pm
Found the best code, I think: https://github.com/JChristensen/Timezone (https://github.com/JChristensen/Timezone)
it is fully customizable.
But since it uses TimeLib there is a conflict

dayOfWeek is used in TimeLib and RTClib but is a define in TimeLib
Code: [Select]
#define dayOfWeek(_time_) ((( _time_ / SECS_PER_DAY + 4) % DAYS_PER_WEEK)+1) // 1 = Sunday
and a function in RTClib
Code: [Select]
uint8_t dayOfWeek() const;
I added this in the opened issue in TimeLib library but I am thinking to modify one of the libs and put the modified library in the same folder of the main sketch because dayOfWeek is used often and if I change the main library it will break others sketch.
The main candidate is RTClib since it has only two files, TimeLib instead has many files.

This issue is well known since years but never resolved.

Edit: I am wrong because I used an old RTClib, the new one uses dayOfTheWeek
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 04, 2017, 10:54 am
Software RTC seems to work well!

Main changes
Hardware RTC removed, it uses now a software RTC that is synchronized every 15 minutes to an NTP server.
RTC is in UTC but Display Clock is automatically adjusted for TimeZone and Daylight Saving Time.

Minor changes


Other possible improvements


Some test required since i wrote it this morning.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 04, 2017, 09:25 pm
@ zoomx

Testing your 08:54 am modifications.  

NTP timestamp is wrong in "LOG.TXT" on Sketch startup, if NTP packet is dropped, or has not been received yet.  Interval that "ThingSpeak.com" uses is affected in that with no fifteen minute interval data is not sent.  No logtoSD function logging occurs when no fifthteen minute interval occurs.

This why I use a real real time clock.  RTC are readily available online at low cost.  

Real Time Clock is needed.

William

Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 04, 2017, 10:19 pm
You're right! Need to improve NTP code or drop it. I have to test much more!
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 05, 2017, 08:32 am
Now my brain is online again!
I forgot that RTClib can work with an hardware RTC just changing this
RTC_Millis Clock;
into this
RTC_DS3231 Clock;


But there is a difference. To have Timezone and DST working, RTC must be in UTC.


I found the NTP client problem here

Code: [Select]
sendNTPpacket(timeServerIP); // send an NTP packet to a time server 
// wait to see if a reply is available
  delay(1000);

  int cb = udp.parsePacket();

I need to change the delay into a code that wait for the response with a timeout.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 05, 2017, 09:10 pm
Wemos D1 R2 is back 100%, online now.  Attaching Sketch that the Wemos D1 R2 server is currently running.

zoomx is doing some modifications; at this time, I have not added his modifications.  Top priority, on project is for downloads to be uninterrupted by events such as logging.  Not so much concerned about missing a request to be parsed.  Response time is quick and the client closes when a request finishes.

Try a download, think you will be amazed at the speed, compared to before the "readFile" function that martinayotte of ESP8266 Community Forum created.

June 29, 2017, I submitted this project to "Create Arduino Project Hub." for review and publication.

Project web pages are back online! (http://tinyurl.com/Observations-weather)

Now featuring four graphs from "ThingSpeak.com:"  Graphed Weather Observations (http://tinyurl.com/Four-Graphs)  Viewable on one web page; trace along the graphed line with your mouse for date/time stamp and value of graphed point.


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 17, 2017, 02:56 pm
Attached there is a test sketch to evaluate the time taken to get a response from an NTP server. I get very short times using the USA NTP pool time.nist.gov

Using the server without hardware RTC is simpler but I think that I can add another define since the switch between hardware and software RTC is very simple, only one line.

It seems that the sketch fit on a cheap ESP-01 module (leaving 64k for SPIFFS) but I have to test it, I have only compiled.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 17, 2017, 06:39 pm
@zoomx

I tested your NTP code[ attached the result.

Do I understand correctly that you have to continuously hit the time server to get the fifteen minute interval?  IMHO, that  is a high demand on a time server.

Loop function is only ran, if on the fifteen minute interval; otherwise, Listen function is listening for a request from a client.

Planning to stay with a Real Time Clock for fifteen minute interval, new day, and day of week; to me, it eliminates high demand on a time server.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 17, 2017, 07:15 pm
The time request is done only one time every 15 minutes in your sketch modified by me, it can be changed, for example around two AM when in EU there is the  daylight time switch.

Instead the test show that you never get a response from the time server. Maybe UDP traffic is firewalled?
Test ask for a response and, if it don't get a response within one second make another request. This is repeated 4 times. You get continuous request only because this is a test.



Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 17, 2017, 07:58 pm
@ zoomx

Re-ran test; Gateway was set to only pass TCP, my apologies.  Attached find test run.

Willing to try your code and run it for a couple of days.  Are you okay with me doing this?

Using a NTP server is a new experience for me; will need to get up to speed.  Thank you.

Attached latest code changes to project as of 7/17/2017.


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 18, 2017, 02:36 am
@ zoomx

How often can 0 seconds be detected by NTP server?  I have been observing Serial output of your "SoftRtcAndNTPusingTimezoneTimeout.ino" and do not see 0 seconds being detected.  If there is no repeatability of 0 seconds; what happens is multiple sets of data gets logged and gets sent to "ThingSpeak.com"  until the next minute occurs.  Solution?

Your code:

Code: [Select]
//Write Data at 15 minute interval  
     if ((minute) % 15 == 0){


This code had to be changed:

Code: [Select]
//Write Data at 15 minute interval  
     if (((minute) % 15 == 0) && ((second) == 0)){



William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 18, 2017, 08:06 am
I don't check for seconds because there is the possibility that at second zero the server is busy. If it is not busy it will send data to thinghspeak at seconds zero. If it is busy, seconds will be not zero but, for example 20. If I check for senconds == 0 and in this second the server is busy, the data will not be sended to thingspeak because when the server will be free the seconds will be not zero.
So when
Code: [Select]
((minute1) % 15 == 0)
data is sended to thingspeak.

Then there is a flag, AcquisitionDone, that became true after sending the data to thingspeak.
The next time, if minute is still ==0, this 
Code: [Select]
((minute1) % 15 == 0)

is true but this

Code: [Select]
(AcquisitionDone == false)

is false because now AcquisitionDone is true.
So no data is sended to thingspeak again.
When

Code: [Select]
((minute1) % 15 == 0)
became false, AcquisitionDone became false too so the next time that

Code: [Select]
((minute1) % 15 == 0)
will be true, AcquisitionDone is false and data is sended to thingspeak.
I wroted this some months ago for an SD datalogger because I have some of them and I want to have all data synchronized.
This send data only every 15 minutes and synchronize RTC with NTP only every 15 minutes too



Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 19, 2017, 02:31 am
@ zoomx

Thank you.

Testing your "Observations_SPIFFS_20170704.ino" I have been unable to open web pages; /Weather, /SdBrowse, or /Graphs.  Have you experience this in your testing?  Is there a conflicting library?  

"Observations_SPIFFS.ino" uses ESP8266Wifi library for both client and server, without your modifications; /Weather, /SdBrowse, and /Graphs web pages open.  I even tried with just listen function in loop; still your version does not open web pages.  

Do you have a later version for testing?

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 19, 2017, 04:13 pm
You have to add the FAVICON.ICO file into SPIFFS. Without this file you can't reach any other page. You can see this in the serial debug.
Just tested, I can see all pages.


I have to add the last update that you wrote.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 19, 2017, 08:46 pm
@ zoomx

All is well...  added delay(500); before listen function call in loop; may need tweaking.

Unzipped your "Observations_SPIFFS_20170704.zip" file; made local port, 123. 

All web pages opening as you stated.  Thank you zoomx.


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 20, 2017, 06:18 am
Testing modded  "Observations_SPIFFS_20170704.zip" with my latest update;  Serial output attached, check out between 7/19/2017 @ 23:00 and 7/19/2017 @ 23:15. packet was received; however, "Seconds since Jan 1 1900 = 0"   Should this condition be tested?  Causes timestamp to be wrong for log file.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 20, 2017, 08:40 am
Message received, I have to check what happens if you don't get response from NTP or get an invalid response to avoid to set the RTC to a wrong time.
Looking at code, if no packet is received there is no update time. I have to add another check to the packet length, it must be 48.
In Wikipedia
https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation (https://en.wikipedia.org/wiki/User_Datagram_Protocol#Checksum_computation)
I read that UDP packet has a checksum so I believe that the packet received should be correct. If packet is not correct it should be discarded and it's like not receiving any packet.
Maybe it can happen that the server is wrong or something happens. Long time ago it happened that GPS went wrong and an automatic GPS data collection didn't start because of this strange error.
If you vave frequent NTP errors it's better that you mantain the DS3231 RTC, as you can see it's only a line different.


Anyway Thingspeak uses it's time server and you can download data from it. In my opinion Thingspeak is great since you can build your own server using a PC or a little RaspberryPI so if they change idea (pachube, cosmic now Xively changed idea!) you can have your server.
Roberto
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 24, 2017, 03:41 am
Thank you zoomx of Arduino.cc Forum for implementing NTP with Timezone for the project!  Tested for two days; time intervals , logging and ThingSpeak are close to working correctly.  Perhaps some small tweaking remains.  Well done zoomx.

Project web page (http://tinyurl.com/Observations-weather)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 24, 2017, 11:25 am
Thanks but I made only a small contriubution!
I thought to use NTP because I am lazy and I prefer a clock that regulate itself. Two time a year I have to regulate many clock when there is the switch to the Daylight Saving Time!
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 25, 2017, 04:49 pm
Two days of testing NTP with Timezones, I am convinced it is not redy for prime time.  Not sure if it is a library or NTP, or a hardware issue.  Server randomly losses track of NTP timing interval.  I am reverting back to using a DS3231 RTC.

Wondering if it could be the ESP8266 going into a sleep mode and not getting NTP packets.

Returned to Real Time Clock version; attached file is current running the Wemos D1 server.

Four-Graphs from Wemos D1 server (http://tinyurl.com/Four-Graphs)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jul 27, 2017, 11:22 am
Testing. Data will be here
https://thingspeak.com/channels/306806 (https://thingspeak.com/channels/306806)
Since it is at work I cannot have a public access to the server. I will have it when I will move it to my home.
This is  just a test for RTC and logging.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 29, 2017, 11:50 am
Good morning,

Re-visited zoomx's application of "NTP with Timezones  and have a successful working sketch, "NTP_Observations_SPIFFS.ino"

Included in the attachment are the sketch, "LOG.TXT (portion actually logging during testing) and output of the Serial Monitor (during testing.)

Note:  Time keeping is now accomplished with software using NTP time server.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Aug 04, 2017, 05:57 am
Server and sensor running "NTP_Observations_SPIFFS.ino" with No Real Time Clock!

Server has been online since 7/29/2017; all time keeping functions are handled by NTP time server.
Thank you "zoomx" for applying NTP and Timezones to project!

Check out thefour graphs from "ThingSpeak.com;" let mouse hover over data point on graph to reveal value and time of data entry:  Four Graphs (http://tinyurl.com/Four-Graphs)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Aug 11, 2017, 10:43 am
I read that UDP packet has a checksum so I believe that the packet received should be correct. If packet is not correct it should be discarded and it's like not receiving any packet.
I am wrong and this assumption seems false.

In Serial output --modded file.tx
Code: [Select]

packet received, length=12
Seconds since Jan 1 1900 = 0
Unix time = 2085978496


Data written to log  02/07/2036 , 01:30:00

there was a packet that was long 12 bytes instead of 48.
So packet with lenght different from 48 must be discarded.

I apologize William, you wrote this already.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Aug 11, 2017, 10:20 pm
@ Zoomx

No need to apologize zoomx.  Thank you for all your time, coding, and community participation.

Logging of Data now has priority after too many missed 15 -minute intervals.

This version has been tested over three days; much improved, over previous versions.  All time keeping is  NTP based; no Real Time Clock!

Project web link --Graphed Observations (http://tinyurl.com/Four-Graphs)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Aug 17, 2017, 05:46 am
Project web pages (http://tinyurl.com/Four-Graphs)

3 Minute Video of project (https://www.youtube.com/embed/sV2-WNSYbzo)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Aug 31, 2017, 06:29 am
Current version has been online since 08/08/2017  @ 23:31 EST with little interruption.  Developement board is a RobotDyn Wifi D1 R2 from Source of board (http://robotdyn.com)

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=224224)

Project web page (http://tinyurl.com/Four-Graphs)


Attached "NTP_Observations_SPIFFS.zip" is the 08/08/2017 @ 23:31 EST version.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Oct 02, 2017, 11:24 pm
October 2, 2017 Update:

"NTP_Observations_SPIFFS.ino"    08/08/2017  @ 23:31 EST continues to remain online.  Last version change date was 08/08/2017 @ 23:31 EST. 

Current version has performed above and beyond my expectations.  Very good reliability.  Fast downloads.  Fast Watchdog resets on the rare occurrence when one happens.


William

Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Oct 30, 2017, 10:50 pm
Improved Wake time in response to page request. 

Improvements to simplify editing page links.

List of line numbers requiring editing.


Project webpage --Observations (http://tinyurl.com/Observations)

Project webpage --Graphs (http://tinyurl.com/Four-Graphs)
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Nov 23, 2017, 06:31 pm
Update 11/23/2017

Added code for occasion when no NTP Packet is received, to request a new Packet to be sent.

Added SERVER.TXT file for date and time of Server starts.  

Previous version of project code is:  "Nov_23_2017__NTP_Observations_SPIFFS.ino."


Project web page (http://tinyurl.com/Observations-weather)

Project's Graphing capabilities (http://tinyurl.com/Four-Graphs)


Installed version of project code is:  "Dec_9_2017__NTP_Observations_SPIFFS.ino."

Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Dec 31, 2017, 10:40 am
Some how my device was blocked at my Gateway.

Apologies to anyone that tried to connect and was unable to connect.

No changes to software since the last update.

Project website (http://tinyurl.com/Observations-weather)


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jan 07, 2018, 11:58 pm
New feature:  Hosted website gets updates every 15 minutes.  Very little additional code to project by using ESP8266HTTPClient library.

Website is on ".000webhostapp.com" free, educational Domain:
Hosted project website. (https://observations-weather.000webhostapp.com/)

Stephan Borsay's  article on Hackster.io (https://www.hackster.io/detox/send-esp8266-data-to-your-webpage-no-at-commands-7ebfec) was the inspiration for this additional feature.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Jan 10, 2018, 11:06 am
Thanks for the update William,
I will read the links you provided!
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jan 24, 2018, 02:46 pm
Completed design of Website for project.  Driving element of web sit is the following code:

Code: [Select]
<?php

/*
  ESP8266: send data to your Domain 

  Uses POST command to send BME280 data to a designated website
  The circuit:
  * BME280
  * Post to Domain 
  
  Original version of this code by Stephan Borsay
  Modified with HTML and CSS by techno500 for use with BME280 along with Arduino/ESP8266 sketch:  
  "NTP_Web_Interface.ino."
  
*/

date_default_timezone_set("America/Indianapolis");
$TimeStamp date("Y-m-d   h:i:sa");

   if( 
$_REQUEST["fahren"] ||  $_REQUEST["heat"] ||  $_REQUEST["hum"] ||  $_REQUEST["dewpt"]
       ||  
$_REQUEST["cpressure"] ||  $_REQUEST["bars"] ) 
   {
      echo 
" The Temperature is: "$_REQUEST['fahren']. "F. <br />";
      echo 
" The Heat Index is: "$_REQUEST['heat']. " Heat Index <br />";
      echo 
" The Temperature is: "$_REQUEST['hum']. " hum <br />";
      echo 
" The Dew Point is: "$_REQUEST['dewpt']. " Dew Point<br />";
      echo 
" The Barometric Pressure is: "$_REQUEST['cpressure']. " in. HG. <br />";
      echo 
" The Barometric Pressure is: "$_REQUEST['bars']. " millibars <br />";
      echo 
" The Altitude is: 824 Feet <br />";
 
   }
  
 
$var1 $_REQUEST['fahren']; 
$var2 $_REQUEST['heat'];
$var3 $_REQUEST['hum'];
$var4 $_REQUEST['dewpt'];
$var5 $_REQUEST['cpressure'];
$var6 $_REQUEST['bars'];
//$var7 = $_REQUEST['altit'];

$WriteMyRequest=

"<html>".
"<head>".
"<body>".
"<style>"
"body { background: #561C0E no-repeat fixed center;}".
"a,h2,h3 {color: #ffffff;}"
"font-weight: bold;}".
"</style>"
"<title>Observations</title>".
"<h2>".
"Observations".
"<p>Treyburn Lakes<br />".
"Indianapolis, IN<br />".
"46239</p></h2><br />".
"<p><h3>Last update:   " $TimeStamp "</p>".
"<p> The Temperature is : "   $var1 " F. </p>".
"<p>And the Heat Index is : " $var2 " F. </p>".
"<p>And the Humidity : "       $var3 " % </p>".
"<p>And The Dew Point is : "   $var4 " F. </p>".
"<p>And The Barometric Pressure is : "  $var5 " in. HG. </p>".
"<p>And The Barometric Pressure is : "  $var6 " mb</p>".
"<p>And The Altitude is : 824 Feet </p><br/><br/>".  // Altitude as read by GPS for project location
"<a href='https://pathToAbout.php'>About</h3></a>".
"</head>".
"</body>".
"</html>";

file_put_contents('display.html'$WriteMyRequest);

?>


This is the PHP code that creates the "Observations" menu option of the website.

Hosted project website (https://observations-weather.000webhostapp.com/)

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=242691)

PHP code above receives the POST from the "NTP_Web_Interface.ino" sketch.  POST is developed with use of the "ESP8266HTTPClient" library.  Project is unique in the fact that it does not use a Real Time Clock.  Every 15 minutes; PHP code rewrites the "Observations" HTML page.  Only the "ESP8266" development board and the BME280 are required making for a real low cost project!  All timing requirements are provided by using Network Time Protocol and a time server to derive the time and date used in the sketch.

Over the time I have been developing this project; have had allot of help from members of the various forums.  Without their help this project would have been impossible.  To those that helped and the owners of the various forums; thank you, your help most appreciated!

Attached is the sketch and the php code.

William



Add screen capture
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: rxpc on Feb 06, 2018, 11:34 pm
Nice project Techno!!!

That BME280 sensor is really nice! I noticed on your observations page the temp is off the scale:

Treyburn Lakes
Indianapolis, IN, US

Last update: 2018-02-06 05:30:02pm

The Temperature is : -230.62 F.

And the Heat Index is : -228.32 F.

And the Humidity : 100.00 %

And The Dew Point is : -230.62 F.

And The Barometric Pressure is : 35.13 in. HG.

And The Barometric Pressure is : 1189.80 mb

And The Altitude is : 824 Feet
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Feb 07, 2018, 01:43 am
@rxpc  Thank you for bringing this to my attention; caused by faulty Dupont jumper on the BME280 sensor.  I need to get this project off of breadboard and on to a printed circuit board.  Have started "Eagle" class on "Udemy.com" for creating printed circuit boards.  First project in "Eagle" will be to create a PCB for "NTP_Web_Interface.ino" project!

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: rxpc on Feb 10, 2018, 01:38 am
Great!  I'm glad you replied...I was wondering if the ice age was upon us!
Good luck with your class.

I made some PCB's a few years back with a laser printer. They make a nice small
footprint for your projects.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Mar 10, 2018, 12:10 am
Weather Data (https://preview.tinyurl.com/Indpls-weather)  is displayed and updated every fifthteen minutes,  Data is pulled from server running on RobotDyn WiFi D1 R2.  Pages "live" on a domain, website.

Full features of project: (http://tinyurl.com/Observations-weather)  Data is displayed and updated every fifteen minutes, File browser and file downloads of collected, weekly Data files and associated files.  Data is collected every fifthteen minutes.  Web site "lives" in RobotDyn WiFi D1 R2 SPIFFS, flash memory.

Both websites are running simultaneously from the same sketch!  Sketch does not require a Real Time Clock!  NTP Protocol and time server are used for all timing requirements!

See previous POST for "NTP_Web_Interface.zip"  sketch and PHP file that receives data from the server on the Domain, hosted web site.



Correct filename.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Apr 12, 2018, 12:34 am
schufti of the ESP8266.com Forum introduced me to two libraries for the ESP8266; "sys/time.h" and "time.h"  these two libraries are used with NTP Time server to sync time with the ESP8266.  Should be noted this requires ESP8266 core 2.4.1 version; which can be installed using the "Arduino IDE" "Board Manager."

I am having issues with the hosted web site; once this is resolved, the website should be back online.

Project, hosted web site (http://https:tinyurl.com/Indpls-weather)

"NTP_Web_Interface code has been renamed to "Fast_Time_Web_Interface_v4.ino."   Current date of version is 04/11/2018 @ 17:03 PM.

Website running RobotDyn WiFi D1 board costing less than $5.00 can be viewed here:

Project web site (http://tinyurl.com/Observations-weather)
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: zoomx on Apr 12, 2018, 08:32 am
I will check these libraries!

Thanks William!

Edit: Wow! seems much simpler than using UDP, TimeLib and TimeZone libraries.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Apr 14, 2018, 09:38 pm
Update 04/14/2018:

Faster performance using the two new libraries , have removed some code related to libraries no longer used.  Everything is working:  ESP8266 server website and the hosted website are both online!

Current version of the "NTP_Web_Interface.ino" has been renamed to:  "Fast_Time_Web_Interface_v5.ino."

More than a Data Logger; also allows fast, remote downloading of files stored onboard in the SPIFFS memory, files are browser enabled and selectable from files list.  Both ESP8266, local server and hosted web server are available from the same sketch.  Commented out are two useful features; a LCD display function and a Audio beep to alert for a predetermined difference in a 15 minute interval of barometric pressure (experimental).

Same board that is online has been used for development of project coding; therefore, there may be gaps in the data logs!  The 15 minute interval has been "spot on" every 15 minutes from the top of the hour.  Awesome, for no real time clock.   All time related requirements are provided by NTP time server.   Even has a backup time server!

Two websites;one sketch:  Web and File browser (https://tinyurl.com/Robotdynwifi) and
Hosted web site; no file browser (https://tinyurl.com/Indpls-weather)  

Restrictions from free hosting service is why there is no file browser.  Graphing of four weather parameters are availble on both websites.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Apr 21, 2018, 12:48 pm
04/21/2018

Removed all occurrences of "\r\n" from lines with "client.println." Implemented your suggestion to use formatted strftime, added %Z for timezone. Works perfectly, thank you schufti! Eliminated most of the timeStamp function except the formatting of "strftime."

File name was changed from "NTP_Web_Interface.zip" to reflect; faster, client connections under one second consistently .

ESP8266 Server web sites: ESP8266 Server Web site (https://tinyurl.com/Robotdynwifi) and  Hosted Web site (https://tinyurl.com/Indpls-weather)  Both web sites obtain data from "Fast_Time_Web_Interface_v7.ino".

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 06, 2018, 08:55 pm
Due to circumstances beyond my control; project has been offline since 05/29/2018.  Online operation was restored 06/06/2018 at 01:30 EDT.  ISP changed from Gateway -based firmware to Gateway cloud -based firmware; there were issues with port forward not working.  Gateway was swapped out for a new one in the troubleshoot process and a new public IP was attached nullifying previous links to the ESP8266 Server web site.

I apologize for any inconvenience experienced by this change.

Active project websites:

ESP8266 Server website --new link (https://tinyurl.com/Treyburnweather)

Domain, hosted website --unchanged (https://tinyurl.com/Indpls-weather)

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jun 24, 2018, 12:49 pm
Experienced sensor failure; awaiting replacement sensor.  Both websites are affected; updated data is invalid!

Affected websites:  

ESP8266 Server (http://https:/tinyurl.com/Treyburnweather)

Domain, hosted website (https://tinyurl.com/Indpls-weather)


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Jul 29, 2018, 02:51 am
Update 7/28/2018

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=267953)

Replacement BME280 installed.  Assembled new sensor board with a 6 pin, female header; so I can use the sensor board to test BME280 sensors, without needing to soldering sensor onto the SB400 prototyping board.

Project has been running the last week reliably and with valid data.  Made some tweaks to the code.  Have observed occasional delays in responding to client requests; does not happen often, when it does happen page has to be reloaded by the Client Browser.  Issue may be the ESP8266 going to default modem sleep mode.   I have added in the loop:

Code: [Select]
wifi_set_sleep_type(NONE_SLEEP_T);
  delay(100);


Trying to keep the ESP8266 from going into default modem sleep --power is not ain issue.  Being occasional and random it is making it difficult to find and to correct.

Cable provider has "refreshed" the WAN IP. (https://tinyurl.com/500-weather)  ESP8266 Server Website.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Aug 09, 2018, 05:52 am
ESP8266 Hosted web site. (https://tinyurl.com/500-weather)

There are two web sites; one hosted on the ESP8266, and one domain, hosted --both are driven by one sketch!

Domain, hosted web site. (https://tinyurl.com/Indpls-weather)

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=269662)


08/08/2018 Current version of sketch is attached.
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Sep 04, 2018, 06:34 am
Update 09/03/2018:  "NTP_Time-sycned_Web_Interface.ino"  has been online without resets of any kind since 08/13/2018; current code is attached in the previous message. No wireless drops or wireless interference.

Development board running web server is a "RobotDyn WiFi D1 R2 at a  low-cost, of under $5."  Board is available at RobotDyn.com; photograph of this board can be viewed in previous "Update of 07/28/2018."

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Feb 19, 2019, 12:13 pm
02/19/2019

New WAN ip address invalidated previous URL links for project ESP8266 server:

  New URL link for ESP8266 project server (https://bit.ly/2DDV5zV)

  Domain Hosted website (https://bit.ly/2M5NBs0)


Fixed FTP transfers from within running sketch for easier admin maintenance of files.

William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Feb 24, 2019, 03:43 am
Created video of Serial Monitor, ESP8266 based server, and Chrome Browse; plus displays of text files.

Video of Project


(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=296261)


"Hosted Web site files.zip" attachment contains php file to be uploaded to a hosted web site to fully use sketch "NTP_Time-Synced_Web_Interface.zip"

William (https://youtu.be/B_iMvUM-PHw)
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Mar 19, 2019, 02:07 am
03/18/2019

Back in October last year I was asked if I could add a tipping rain gauge and GPS to the project.  After months of endless ESP8266 wdt resets and my limited skills of debugging; I put the addition on the shelf.  Months later I purchased ESP32 Dev. Edtion Board (https://www.adafruit.com/product/3269).

I was excited to learn the ESP32 has 2 cores; one for wifi operations and one for processing.  My project has "allot of moving parts" or so I have been told.  To make a long story short.  Using the code for from Haroon552; we have impemented a tipping Rain Gauge and a NEO m8n GPS module to the project.  Armed with the ESP32 and ESP32 libraries I began work again the addtion to the project.  This time I experienced no wdt resets.  Project with the requested addtions have been online for a few days now with no wdt resets.

"Weather_View.ino" Rain Station with GPS and RTC (https://bit.ly/2FiuNE5)  Project website

Once finished with addtional testing I will post the "Weather_View.ino" file.

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=299926)

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=299928)


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Mar 24, 2019, 04:58 pm
03/24/2019

ESP32 "Weather_View" project web site. (https://bit.ly/2FiuNE5)

Have finished work on "Weather_View.ino."  Features added include Rain Gauge support, GPS Longitude, GPS Latitude, GPS Altitude, GPS Time, File Transfer using FTP (Filezilla), and Over the Air (OTA) updates. 

Existing features of "NTP_Time-Synced_Web_Interface.ino" kept; except for NTP time keeping.

GPS time is used to set the DS3231, Real Time Clock; which can be set at predetermined interval.

"NTP_Time_Synced_Web_Interface.ino" has been renamed "Weather_View.ino."

Zipped file includes "dataCollector.php" to be uploaded to a hosted web site for using webInterface function of sketch.

Project used a ESP32 Dev. Edtion Board, purchased from "Adafruit.com."  GPS Module used is a "NEO m8n."

Rain Gauge code  and EEPROM routines were developed by Haroon552 of this forum.


William
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on Apr 24, 2019, 07:02 am
"NTP_Time_Synced_Web_Interface.ino" hosted domain website has been visited by users from 67 Countires, thank you for all the visits to the hosted domain website of the project!

Hosted Domain website (https://bit.ly/2M5NBs0)

(https://forum.arduino.cc/index.php?action=dlattach;topic=466867.0;attach=305305)

William

Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Bringamosa on Apr 29, 2019, 05:37 pm
Awesome :)
Title: Re: ESP8266 --Internet Weather Datalogger and Dynamic Web Server
Post by: Techno500 on May 01, 2019, 07:30 am
Thank you Bringamosa for the feedback!

I have re-uploaded the current project code.  There are two files to the project:  "NTP_Time-synced_Web_Interface.zip" and "Hosted Web site files.zip."

Credit for PHP files and concept of sending data to webpage goes to: Stephen Borsay's project (https://www.hackster.io/detox/send-esp8266-data-to-your-webpage-no-at-commands-7ebfec)  Project PHP file is in "Hosted Web site files.zip."


William