Project Guidence - Home Automation/Datacollection

Ok....So I am at the very start of a project that I want to do for fun and learning.....

In very simple terms what I was to do is not really home Automation but more of a data collection from various sensors dotted around the Home and garden onto a PC/Server and display in graphics format and general data analysis. The idea is to make it modular so that it can be expanded/added to over time.

So I am at the very early stage of the project i.e Pen and paper and searching for ways to do it, and kind of getting a high level idea of how to approach it, but I want to spend as much time at this stage, because I want to expand this in the future, I want to future proof it as much as possible, or at least make it easy to expand rather than having to redesign in order to expand

So below is a basic diagram of what I am thing about.
There will be a PC for Data collection - PC will have webserver, SQl and probably Python
There will be several Arduinos dotted around to collect the data, possibly some with IO extender w/I2C to collect a lot of data from some locations.
Communication between the PC and Arduino's will need to be Wireless (Not WIFI) due to distance some will be from the PC - Distance will be less than 100m, but some will be inside and some outside.
Arduinos will probably have local storage (SD card) to also record data and prevent any data loss. - This might be overkill
There will probably be some kind of time syncing (NTP) with the PC to ensure all data has the same timestamp.

Now what am I looking for :slight_smile:

Well Ideas to a couple of questions. I have been searching about and can find different ways of doing what i want, but i am interest to hear other people thoughts, particularly forsome pitfalls or things to be aware of....

What could be a good Wireless option for communication? I found this link to be a good source on information and I am leaning towards RF, but are there other or better options? Any reason to not use RF?
https://www.sparkfun.com/pages/wireless_guide

What would be the best way to collect the data?
Push from the Arduinos or Pull from the Arduinios? Pitfalls with either?
What checks could be put in place to sure data quality?

If I was to incorporate the SD card for local data collection - How do I ensure data is collected correctly from the arduino and is not duplicated or missing data?
the way i have this in my head, is that the Arduino would read the data from the sensor, write it to the SD card.
Then when it comes to transfer the data to the SD card how can you ensure that it starts the transfer at next piece of data to be transferred so that it is not something already sent?
For some reason I cannot get my head around this.......And i am probably overthinking it...

Some practical advice....to use an Arduino outside, in damp environment. What is the best way to protect it?

I have 13 ESP32's around the property. All sending and receiving data via a Raspberry Pi doing the MQTT Broker thing. A Python program on the RPi does the ESP32's to dB stuff. All wireless.

I can go to my website, set the temp of the HVAC, and have the ESP32 running in place of the HVAC controller make the changes.

. There are 2 ESP32's in that box doing the weather thing and other chores.

On top of the box is a pagoda thingy. That's a Steven's Shield. Inside the shield is a ESP32 and a BME680. The BME 680 is exposed to the weather. And it stays cozy in the harsh weather

1 Like

Hi @Keeks
I recommend instead of using arduin, use either ESP8266 or ESP32 as it has native wifi.

As it will be a data collection network and in the future to control devices, I recommend the use of Mesh network, as this way the scope of physical space is extensive, even reaching distances of 1Km.

RV mineirin

1 Like

Thanks Guys.....This is what I was looking for - different ideas of how to approach this and widen my search. I never realised ESP had native WIFI so I will look into MQTT and Mesh Network.

My main recommendation is avoid using anything "in the cloud" as you are likely over time to invest a lot of time and effort building a system and you don't want to discover a few years down the line that everything stops working for reasons out of your control.

I agree, ESP32 is ideal for this sort of thing.
For sensors which are only going to trigger occasionally (e.g. shed door) I just power one directly from an 18650 cell using a three wire reed switch which turns the power on then have the esp32 do it's thing then turn its own power off via a relay or transistor - this way the battery can last literally years on a single charge.
btw - I know the voltage can be outside the official tolerances but so far I have not had any issues with this.

The thing is just get something basic up and running then over time as you learn more and have more ideas you can add to it / improve it.
I think MQTT is the way to go but this did not exist when I started building my home automation (25 years back!).

1 Like

It's certainly risky to have your system rely on cloud services, especially for core operations. I pull weather data to display and I'm on my third provider now after the first two got bought out and promptly canceled their free tiers.

It is nice though to have a way to talk to your system from your phone without having to poke a hole in your firewall and cloud services can give you that e.g. Blynk. If you're really worried about the services you use going away, you can always configure your own cloud servers and cloud balance across the major providers but that's a bit extreme for home automation.

It raises the design question though about what to do in a power outage. Initially, I'd guess, nothing, but the OP does seem to be considering battery powered operations continuing to collect data and sending it to the PC when it comes back up. Consider too, how you'll manage anything that's automated if the PC has a hardware failure.

I'd be inclined to have that PC be a standalone rather than your daily driver. I'd also want it to be a machine that I can control updates on so that it doesn't take it on itself to reboot when it feels the need. A Pi probably.

1 Like

Oi! So close. MQTT was invented in 1999.

MQTT is available for free. A RPi 4 costs about a 100.00 USD.

I did, initially, use AWS as my MQTT Broker. AWS is free for a year, then the minimum data charge is 1 USD per day. Getting a RPi and setting up my own Broker was cheaper. Also, the RPi, is programable and can do other services, like being an FTP server for images, running TensorFlow for images recognition, and does not close up shop on a whim.

1 Like

This is one of the main things in my initial design that I want to plan for......If a sensor is able to read data at its location, then it should be able to stored it locally in case the is a fault with the data collector or a communication fault etc......

One way I was thinking of approaching this was to have 2 sets of data - one live and one historical

The Live data could be published at a regular interval or as requested
The historically stored locally which would be computed at 10/15 min intervals of average/Min/Max and then this could be pushed to the data collector

Now the other thing I am unsure of is using the ESP WIFI - does this need to be connected to my existing WIFI or can it be separate to work?

I found that ESP32's work best on their own network.

Once a second the Python program running on the RPi sends a OK topic that has a payload of date and time. That way the ESP32's do not go to the internet for date and time; keeping them isolated from the internet.

I have 2 ESP32's in the crawlspace under the house and 2 in the attic. When I need to reset those ESP32's. I don't. Instead, I let the ESP32's reset themselves when the RPi is reset for some reason or another. When the ESP32's lose the OK signal for a time, the ESP32's will reset which causes them to get new credentials and then reconnect to the RPi.

Yup you can do that. I found the ESP32's on their own network connected to the RPi is reliable.

On the ESP32 network, only the RPi has access to the internet.

1 Like

Here is some ESP32 Multitasking code that you can use as a model for your project:

#include <WiFi.h>
#include <PubSubClient.h>
#include "certs.h" // include the connection infor for WiFi and MQTT
#include "sdkconfig.h" // used for log printing
#include "esp_system.h"
#include "freertos/FreeRTOS.h" //freeRTOS items to be used
#include "freertos/task.h"
#include <SPI.h>
#include <Adafruit_Sensor.h>
#include "Adafruit_BME680.h"
#include <Adafruit_GFX.h>    // Core graphics library
#include <Adafruit_ST7789.h> // Hardware-specific library for ST7789
#include <driver/adc.h>
#include "esp32-hal-ledc.h"
#include <HardwareSerial.h>
#include <SimpleKalmanFilter.h>
#include "MHZ19.h"
#include <ESP32Time.h>
#include <SolarCalculator.h>
////
ESP32Time rtc;
MHZ19 myMHZ19;
////
Adafruit_BME680 bme( GPIO_NUM_5 ); // use hardware SPI, set GPIO pin to use
//Adafruit_ST7789 tft = Adafruit_ST7789( TFT_CS     , TFT_DC    , TFT_MOSI   , TFT_SCLK   , TFT_RST     );
Adafruit_ST7789 tft   = Adafruit_ST7789( GPIO_NUM_15, GPIO_NUM_0, GPIO_NUM_13, GPIO_NUM_14, GPIO_NUM_22 );
WiFiClient   wifiClient; // do the WiFi instantiation thing
PubSubClient MQTTclient( mqtt_server, mqtt_port, wifiClient ); //do the MQTT instantiation thing
//////
#define evtDoParticleRead     ( 1 << 0 ) // declare an event
#define evtWaitForBME         ( 1 << 1 )
#define evtParseMQTT          ( 1 << 3 )
EventGroupHandle_t eg; // variable for the event group handle
//////
QueueHandle_t xQ_WindChillDewPoint;
QueueHandle_t xQ_eData; // environmental data to be displayed on the screen
struct stu_eData
{
  float  Temperature = 0.0f;
  float  Pressure    = 0.0f;
  float  Humidity    = 0.0f;
  float  IAQ         = 0.0f; // Index Air Quality
  float  RM0         = 0.0f; // Remaining Moisture from sensor 0
  float  PM2         = 0.0f; // particles in air
  float  WS          = 0.0f; // wind speed
  String WD          = "";   // wind direction
  float  RF          = 0.0f; // rainfall
  float  WSV         = 0.0f; // weather station volts
  float  WSC         = 0.0f; // weather station current
  float  WSP         = 0.0f; // weather station power
  float  WindChill   = 0.0f; //windchill
  float  DewPoint    = 0.0f; //dew point or dew index
  int    SunRiseHr   = 0;    // sunrise hour
  int    SunRiseMin  = 0;    //sunrise minute
  int    SunSetHr    = 0;    //sunset hour
  int    SunSetMin   = 0;    //sunset minute
  int    DuskHr      = 0;    //dusk
  int    DuskMin     = 0;    //dusk
  int    DawnHr      = 0;    // dawn
  int    DawnMin     = 0;    // dawn
  int    TransitHr   = 0;    // 'noon' time
  int    TransitMin  = 0;    // 'noon' time
  double azimuth     = 0.0f;   // Sun's azimuth, in degrees
  double elevation   = 0.0f;     // Sun's elevation, in degrees
} x_eData; // environmental data
QueueHandle_t xQ_Message; // payload and topic queue of MQTT payload and topic
const int payloadSize = 100;
struct stu_message
{
  char payload [payloadSize] = {'\0'};
  String topic ;
} x_message;
////
const float oGasResistanceBaseLine = 149598.0f;
int mqttOK = 0;
int CO2    = 0;
volatile bool TimeSet = false;
//////
esp_timer_handle_t oneshot_timer; //veriable to store the hardware timer handle
//////
SemaphoreHandle_t sema_MQTT_KeepAlive;
SemaphoreHandle_t sema_PublishPM;
SemaphoreHandle_t sema_mqttOK;
////
//serial(2) = pin25 RX, pin26 TX
HardwareSerial co2Serial ( 2 );
//////
// interrupt service routine for WiFi events put into IRAM
void IRAM_ATTR WiFiEvent(WiFiEvent_t event)
{
  switch (event) {
    case SYSTEM_EVENT_STA_CONNECTED:
      log_i("Connected to WiFi access point");
      break;
    case SYSTEM_EVENT_STA_DISCONNECTED:
      log_i("Disconnected from WiFi access point");
      break;
    case SYSTEM_EVENT_AP_STADISCONNECTED:
      log_i("WiFi client disconnected");
      break;
    default: break;
  }
} // void IRAM_ATTR WiFiEvent(WiFiEvent_t event)
//////
void IRAM_ATTR oneshot_timer_callback( void* arg )
{
  BaseType_t xHigherPriorityTaskWoken;
  xEventGroupSetBitsFromISR( eg, evtDoParticleRead, &xHigherPriorityTaskWoken );
} //void IRAM_ATTR oneshot_timer_callback( void* arg )
//////
void IRAM_ATTR mqttCallback(char* topic, byte * payload, unsigned int length)
{
  memset( x_message.payload, '\0', payloadSize ); // clear payload char buffer
  x_message.topic = ""; //clear topic string buffer
  x_message.topic = topic; //store new topic
  int i = 0; // extract payload
  for ( i; i < length; i++)
  {
    x_message.payload[i] = ((char)payload[i]);
  }
  x_message.payload[i] = '\0';
  xQueueOverwrite( xQ_Message, (void *) &x_message );// send data to queue
} // void mqttCallback(char* topic, byte* payload, unsigned int length)
////
void setup()
{
  co2Serial.begin( 9600 , SERIAL_8N1, 25, 26 ); // pin25 RX, pin26 TX
  x_eData.WD.reserve(50);
  x_message.topic.reserve( payloadSize );
  xQ_WindChillDewPoint = xQueueCreate( 1, sizeof(stu_eData) );
  xQ_Message  = xQueueCreate( 1, sizeof(stu_message) );
  xQ_eData    = xQueueCreate( 1, sizeof(stu_eData) ); // sends a queue copy of the structure
  //
  sema_PublishPM = xSemaphoreCreateBinary();
  xSemaphoreGive( sema_PublishPM );
  sema_mqttOK    =  xSemaphoreCreateBinary();
  xSemaphoreGive( sema_mqttOK );
  //
  ledcSetup( 4, 12000, 8 ); // ledc: 4  => Group: 0, Channel: 2, Timer: 1, led frequency, resolution  bits
  ledcAttachPin( GPIO_NUM_12, 4 );   // gpio number and channel
  ledcWrite( 4, 0 ); // write to channel number 4
  //
  eg = xEventGroupCreate(); // get an event group handle
  // output mode
  gpio_config_t io_cfg = {}; // initialize the gpio configuration structure
  io_cfg.mode = GPIO_MODE_OUTPUT; // set gpio mode
  io_cfg.pin_bit_mask = ( (1ULL << GPIO_NUM_4) ); //bit mask of the pins to set
  gpio_config(&io_cfg); // configure the gpio based upon the parameters as set in the configuration structure
  gpio_set_level( GPIO_NUM_4, LOW); // set air particle sensor trigger pin to LOW
  // input mode
  io_cfg = {}; // reinitialize the gpio configuration structure
  io_cfg.mode = GPIO_MODE_INPUT; // set gpio mode. GPIO_NUM_0 input from water level sensor
  io_cfg.pin_bit_mask = ( (1ULL << GPIO_NUM_0) | (1ULL << GPIO_NUM_27)  ); //bit mask of the pins to set, assign gpio number to be configured
  gpio_config(&io_cfg); // configure the gpio based upon the parameters as set in the configuration structure
  // set up A:D channels, refer: https://dl.espressif.com/doc/esp-idf/latest/api-reference/peripherals/adc.html
  adc1_config_width(ADC_WIDTH_12Bit);
  adc1_config_channel_atten(ADC1_CHANNEL_0, ADC_ATTEN_DB_11);// using GPIO 36
  // https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/system/esp_timer.html?highlight=hardware%20timer High Resoultion Timer API
  esp_timer_create_args_t oneshot_timer_args = {}; // initialize High Resoulition Timer (HRT) configuration structure
  oneshot_timer_args.callback = &oneshot_timer_callback; // configure for callback, name of callback function
  esp_timer_create( &oneshot_timer_args, &oneshot_timer ); // assign configuration to the HRT, receive timer handle
  //
  xTaskCreatePinnedToCore( fparseMQTT, "fparseMQTT", 7000,  NULL, 5, NULL, 1 );
  xTaskCreatePinnedToCore( MQTTkeepalive, "MQTTkeepalive", 5000, NULL, 6, NULL, 1 );
  xTaskCreatePinnedToCore( DoTheBME680Thing, "DoTheBME280Thing", 20000, NULL, 5, NULL, 1);
  xTaskCreatePinnedToCore( fDoParticleDetector, "fDoParticleDetector", 6000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fmqttWatchDog, "fmqttWatchDog", 5000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fDoTheDisplayThing, "fDoTheDisplayThing", 23000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fScreenBlanking, "fScreenBlanking", 2000, NULL, 2, NULL, 1 );
  xTaskCreatePinnedToCore( fGetCO2, "fGetCO2", 4500, NULL, 2, NULL, 1 );
  xTaskCreatePinnedToCore( fParseDewPointWindChill, "fParseDewPointWindChill", 4500, NULL, 2, NULL, 1 );
  xTaskCreatePinnedToCore( fSolarCalculations, "fSolarCalculations", 10000, NULL, 2, NULL, 1 );
} //void setup()
////
void fSolarCalculations ( void *pvParameters )
{
  double latitude  = 43.618881;
  double longitude = -116.215019;
  double sunrise;  // Sunrise, in hours (UTC)
  double transit;  // Solar noon, in hours (UTC)
  double sunset;   // Sunset, in hours (UTC)
  double dawn;     // Civil dawn, in hours (UTC)
  double dusk;     // Civil dusk, in hours (UTC)
  //double rt_ascension;  // Sun's right ascension, in degrees
  //double declination;   // Sun's declination, in degrees
  const  float time_zone = -7.0f;
  TickType_t xLastWakeTime = xTaskGetTickCount();
  const TickType_t xFrequency = 1000; //delay for mS
  int count  = 7195;
  int monthX = 1;
  int dayX   = 1;
  for (;;)
  {
    if ( count % 60 == 0 )
    {)
      if ( (rtc.getHour(true) >= 12) & (rtc.getHour(true) <= 23) )
      {
        dayX = 0;
      } else {
        dayX = 1;
        log_i( "ting b" );
      }
      calcSunriseSunset( rtc.getYear(), (rtc.getMonth() + monthX) , (rtc.getDay() + dayX), latitude, longitude, transit, sunrise, sunset );  // Calculate the times of sunrise, transit and sunset
      sunrise += time_zone;
      sunset  += time_zone;
      SolarTimeFormat( sunrise, 0 );
      SolarTimeFormat( sunset, 1 );      
      calcCivilDawnDusk( rtc.getYear(), (rtc.getMonth() + monthX) , rtc.getDay(), latitude, longitude, transit, dawn, dusk); // Calculate the times of civil dawn and dusk (UTC)
      transit += time_zone;
      dawn    += time_zone;
      dusk    += time_zone;
      SolarTimeFormat( dawn, 2 );
      SolarTimeFormat( dusk, 3 );
      SolarTimeFormat( transit, 4 );      
      calcHorizontalCoordinates( rtc.getYear(), (rtc.getMonth() + monthX) , rtc.getDay(),  rtc.getHour() , rtc.getMinute(), rtc.getSecond(), latitude, longitude, x_eData.azimuth, x_eData.elevation );
      x_eData.azimuth   = double(round(x_eData.azimuth * 100)) / 100; // Round to two decimal places
      x_eData.elevation = double(round(x_eData.elevation * 100)) / 100;
      if (count == 7200 )
      {
        SolarTimeFormat( 0.0f, 5 ); // publish MQTT
        count = 0;
      }
    }
    //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
    xLastWakeTime = xTaskGetTickCount();
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
    count++;
  } //for (;;)
  vTaskDelete( NULL );
} //void fSolarCalculations ( )
////
void SolarTimeFormat( double h, int i  )
{
  int hours   = 0;
  int minutes = 0;
  if ( h != 0 )
  {
    int m = int(round(h * 60));
    hours = (m / 60) % 24;
    minutes = m % 60;
  }
  switch ( i )
  {
    case 0:
      x_eData.SunRiseHr = hours;
      x_eData.SunRiseMin = minutes;
      break;
    case 1:
      x_eData.SunSetHr = hours;
      x_eData.SunSetMin = minutes;
      break;
    case 2:
      x_eData.DawnHr = hours;
      x_eData.DawnMin = minutes;
      break;
    case 3:
      x_eData.DuskHr = hours;
      x_eData.DawnMin = minutes;
      break;
    case 4:
      x_eData.TransitHr = hours;
      x_eData.TransitMin = minutes;
      break;
    case 5:
      String sTopic = "";
      sTopic.reserve( 35 );
      sTopic.concat( String(x_eData.SunRiseHr) + "," );
      sTopic.concat( String(x_eData.SunRiseMin) + "," );
      sTopic.concat( String(x_eData.SunSetHr) + "," );
      sTopic.concat( String(x_eData.SunSetMin) + "," );
      sTopic.concat( String(x_eData.DawnHr) + "," );
      sTopic.concat( String(x_eData.DawnMin) + "," );
      sTopic.concat( String(x_eData.TransitHr) + "," );
      sTopic.concat( String(x_eData.TransitMin) );
      sTopic = "";
      xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY );
      MQTTclient.publish( topicSRSSDDT, sTopic.c_str() );
      xSemaphoreGive( sema_MQTT_KeepAlive );
      sTopic.concat( String(x_eData.azimuth) + "," + String(x_eData.azimuth) );
      xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY );
      MQTTclient.publish( topicAzEle, sTopic.c_str() );
      xSemaphoreGive( sema_MQTT_KeepAlive );
      sTopic = "";
      break;
  } // switch ( i ) {
} // void SolarTimeFormat( double h, int i  )
/*
  250-400ppm Normal background concentration in outdoor ambient air
  400-1,000ppm  Concentrations typical of occupied indoor spaces with good air exchange
  1,000-2,000ppm  Complaints of drowsiness and poor air.
  2,000-5,000 ppm Headaches, sleepiness and stagnant, stale, stuffy air. Poor concentration, loss of attention, increased heart rate and slight nausea may also be present.
  5,000 Workplace exposure limit (as 8-hour TWA) in most jurisdictions.
  >40,000 ppm Exposure may lead to serious oxygen deprivation resulting in permanent brain damage, coma, even death.
*/
void fParseDewPointWindChill( void *pvParameters )
{
  while ( !MQTTclient.connected() )
  {
    vTaskDelay( 250 );
  }
  struct stu_message px_message;
  String sDewPoint = "";
  String sWindChill = "";
  sDewPoint.reserve( payloadSize );
  sWindChill.reserve( payloadSize );
  for (;;)
  {
    if ( xQueueReceive(xQ_WindChillDewPoint, &px_message, portMAX_DELAY) == pdTRUE )
    {
      sDewPoint = px_message.payload;
      int commaIndex = sDewPoint.indexOf(',');
      sWindChill.concat ( sDewPoint.substring(0, commaIndex) );
      sDewPoint.remove( 0, (commaIndex + 1) );
      x_eData.WindChill = sWindChill.toFloat();
      x_eData.DewPoint = sDewPoint.toFloat();
      sDewPoint = "";
      sWindChill = "";
    }
    //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
}
////
void fGetCO2 ( void *pvParameters )
{
  uint64_t TimePastKalman  = esp_timer_get_time();
  myMHZ19.begin( co2Serial );
  myMHZ19.autoCalibration();
  TickType_t xLastWakeTime = xTaskGetTickCount();
  const TickType_t xFrequency = 1000; //delay for mS
  SimpleKalmanFilter KF_CO2( 1.0f, 1.0f, .01f );
  for ( ;; )
  {
    KF_CO2.setProcessNoise( (esp_timer_get_time() - TimePastKalman) / 1000000.0f );
    CO2 = KF_CO2.updateEstimate( myMHZ19.getCO2() ); // apply simple Kalman filter
    TimePastKalman = esp_timer_get_time();
    xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY );
    MQTTclient.publish( topicCO2, String(CO2).c_str() );
    xSemaphoreGive( sema_MQTT_KeepAlive );
    // process wind chill and dew point
    xLastWakeTime = xTaskGetTickCount();
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
    //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
} //void fMHZ19B ( void *pvParameters )
////
void fScreenBlanking( void *pvParameters )
{
  int       TimeOfPause = 10000 * 1000;
  uint64_t  PauseStartTime = esp_timer_get_time();
  bool      Pause = false;
  const int brightness = 250;
  int       countUpDown = brightness;
  for ( ;; )
  {
    if (!Pause )
    {
      //if motion detect then show display otherwise blank display
      if ( !(gpio_get_level( GPIO_NUM_27)) )
      {
        for ( countUpDown; countUpDown-- > 0; )
        {
          ledcWrite( 4, countUpDown ); // write to channel number 4, dim backlight
          vTaskDelay( 7 );
        }
      } else {
        Pause = true;
        PauseStartTime = esp_timer_get_time();
        ledcWrite( 4, brightness );
        countUpDown = brightness;
      }
    } else {
      // still detecting movement reset blanking pause time
      if ( gpio_get_level( GPIO_NUM_27) )
      {
        PauseStartTime = esp_timer_get_time(); // extend pause blanking time
      }
      if ( (esp_timer_get_time() - PauseStartTime) >= TimeOfPause )
      {
        Pause = false;
      }
    }
    vTaskDelay( 250 );
  }
  vTaskDelete( NULL );
} //void fScreenBlanking( void *pvParameters )
//////
void fparseMQTT( void *pvParameters )
{
  struct stu_message px_message;
  for (;;)
  {
    if ( xQueueReceive(xQ_Message, &px_message, portMAX_DELAY) == pdTRUE )
    {
      xSemaphoreTake( sema_mqttOK, portMAX_DELAY );
      mqttOK = 0;
      xSemaphoreGive( sema_mqttOK );
      if ( px_message.topic == topicRemainingMoisture_0 )
      {
        x_eData.RM0  = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicWindSpeed )
      {
        x_eData.WS = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicWindDirection )
      {
        x_eData.WD = "";
        x_eData.WD = String(px_message.payload);
      }
      if ( px_message.topic == topicRainfall )
      {
        x_eData.RF = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicWSVolts )
      {
        x_eData.WSV = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicWSCurrent )
      {
        x_eData.WSC = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicWSPower )
      {
        x_eData.WSP = String(px_message.payload).toFloat();
      }
      if ( px_message.topic == topicDPnWI )
      {
        xQueueSend( xQ_WindChillDewPoint, (void *) &px_message, 1 );
      }
      if ( String(px_message.topic) == topicOK )
      {
        if ( !TimeSet)
        {
          String temp = "";
          temp.reserve(50);
          temp.concat( String(px_message.payload[0]) );
          temp.concat( String(px_message.payload[1]) );
          temp.concat( String(px_message.payload[2]) );
          temp.concat( String(px_message.payload[3]) );
          int year =  temp.toInt();
          temp = "";
          temp.concat( String(px_message.payload[5]) + String(px_message.payload[6]) );
          int month =  temp.toInt();
          temp =  "";
          temp.concat(String(px_message.payload[8]) + String(px_message.payload[9]) );
          int day =  temp.toInt();
          temp = "";
          temp.concat( String(px_message.payload[11]) + String(px_message.payload[12]) );
          int hour =  temp.toInt();
          temp = "";
          temp.concat( String(px_message.payload[14]) + String(px_message.payload[15]) );
          int min =  temp.toInt();
          rtc.setTime( 0, min, hour, day, month, year );
          log_i( "rtc  %s Year %d month %d day %d", rtc.getTime(), rtc.getYear(), (rtc.getMonth() + 1), rtc.getDay() );
          TimeSet = true;
        }
      }
    } //if ( xQueueReceive(xQ_Message, &px_message, portMAX_DELAY) == pdTRUE )
  } //for(;;)
  vTaskDelete( NULL );
} // void fparseMQTT( void *pvParameters )
////
void fDoTheDisplayThing( void * parameter )
{
  tft.init( 240, 320 ); // Init ST7789 320x240
  tft.setRotation( 3 );
  tft.setTextSize( 3 );
  tft.fillScreen( ST77XX_BLACK );
  tft.setTextWrap( false );
  struct stu_eData px_eData;
  const int brightness = 250;
  ledcWrite( 4, brightness ); //backlight set
  const int MaxString      = 20;
  String oldTempString     = "";
  String oldHumidityString = "";
  String oldAQIString      = "";
  String oldRainfall       = "";
  String oldWindDirection  = "";
  String oldAirPressure    = "";
  String oldRMO            = "";
  String oldPM2            = "";
  String oldPower          = "";
  oldHumidityString.reserve( MaxString );
  oldWindDirection.reserve( MaxString );
  oldAirPressure.reserve( MaxString );
  oldTempString.reserve( MaxString );
  oldAQIString.reserve( MaxString );
  oldRainfall.reserve( MaxString );
  oldPower.reserve( MaxString );
  oldRMO.reserve( MaxString );
  oldPM2.reserve( MaxString );
  bool Tick = true;
  const int numOfColors = 40;
  /* https://chrishewett.com/blog/true-rgb565-colour-picker/#:~:text=A%20true%20RGB565%20colour%20picker%2021st%20Oct%202017,in%205%20bits%20and%20green%20in%206%20bits. */
  int colors[numOfColors] = { ST77XX_BLACK, ST77XX_RED, ST77XX_WHITE, ST77XX_BLUE, ST77XX_GREEN, ST77XX_CYAN, ST77XX_MAGENTA, ST77XX_YELLOW, 0xd55b, 0xee09,
                              0x2e15, 0xcb43, 0x6bad, 0x126f, 0x1264, 0xe264, 0xe7e4, 0x87e4, 0x87fe, 0x876a,
                              0xe304, 0x1cc4, 0xf4c4, 0xf4da, 0xcf66, 0xa879, 0x7f28, 0x4f37, 0xfa97, 0x6195,
                              0X8162, 0xc962, 0x517b, 0x325b, 0xea5b, 0x179b, 0xff80, 0xf960, 0x416d, 0x7bd1
                            };
  int colorCounter = 1;
  for (;;)
  {
    if ( xQueueReceive(xQ_eData, &px_eData, portMAX_DELAY) == pdTRUE )
    {
      tft.setCursor( 0, 0 );
      tft.setTextColor( colors[0] );
      tft.print( oldTempString );
      tft.setCursor( 0, 0 );
      tft.setTextColor( colors[colorCounter] );
      oldTempString = "";
      if ( Tick )
      {
        oldTempString.concat( "iTemp " + String(px_eData.Temperature) + "F" );
      } else {
        oldTempString.concat( "Wind Chill " + String(px_eData.WindChill) + "F" );
      }
      tft.println( oldTempString );
      tft.setCursor( 0, 30 );
      tft.setTextColor( colors[0] );
      tft.print( oldHumidityString );
      tft.setCursor( 0, 30 );
      tft.setTextColor( colors[colorCounter] );
      oldHumidityString = "";
      if ( Tick )
      {
        oldHumidityString.concat( "iHum  " + String(px_eData.Humidity) + "%" );
      } else {
        if ( (px_eData.SunRiseHr < 10) & (px_eData.SunRiseMin < 10) )
        {
          oldHumidityString.concat( "sRise 0" + String(px_eData.SunRiseHr) + "0" + String(px_eData.SunRiseMin) );
        }
        if ( (px_eData.SunRiseHr >= 10) & (px_eData.SunRiseMin < 10) )
        {
          oldHumidityString.concat( "sRise " + String(px_eData.SunRiseHr) + "0" + String(px_eData.SunRiseMin) );
        }
        if ( (px_eData.SunRiseHr < 10) & (px_eData.SunRiseMin >= 10) )
        {
          oldHumidityString.concat( "sRise 0" + String(px_eData.SunRiseHr) + String(px_eData.SunRiseMin) );
        }
        if ( (px_eData.SunRiseHr >= 10) & (px_eData.SunRiseMin >= 10) )
        {
          oldHumidityString.concat( "sRise " + String(px_eData.SunRiseHr) + String(px_eData.SunRiseMin) );
        }
      }
      tft.println( oldHumidityString );
      tft.setCursor( 0, 60 );
      tft.setTextColor( colors[0] );
      tft.print( oldAirPressure );
      tft.setCursor( 0, 60 );
      tft.setTextColor( colors[colorCounter] );
      oldAirPressure = "";
      if ( Tick )
      {
        //oldAirPressure.concat( "Pres " + String(px_eData.Pressure) + "mmHg" );
        oldAirPressure.concat( "Dew Pt. " + String(px_eData.DewPoint) + "F" );
      } else {
        if ( px_eData.SunSetMin < 10 )
        {
          oldAirPressure.concat( "sSet " + String(px_eData.SunSetHr) + "0" + String(px_eData.SunSetMin) );
        }
        if ( px_eData.SunRiseMin >= 10 )
        {
          oldAirPressure.concat( "sSet " + String(px_eData.SunSetHr) + String(px_eData.SunSetMin) );
        }
      }
      tft.println( oldAirPressure );
      tft.setCursor( 0, 90 );
      tft.setTextColor( colors[0] );
      tft.print( oldAQIString );
      tft.setCursor( 0, 90 );
      tft.setTextColor( colors[colorCounter] );
      oldAQIString = "";
      oldAQIString.concat( "iAQI " + String(px_eData.IAQ) + "%" );
      tft.println( oldAQIString );
      tft.setCursor( 0, 120 );
      tft.setTextColor( colors[0] );
      tft.print( oldRMO );
      tft.setCursor( 0, 120 );
      tft.setTextColor( colors[colorCounter] );
      oldRMO = "";
      if ( Tick )
      {
        oldRMO.concat( "iRM0 " + String(px_eData.RM0) + "%" );
      } else {
        oldRMO.concat( "iCO2 " + String(CO2) + "ppm" );
      }
      tft.println( oldRMO );
      tft.setCursor( 0, 150 );
      tft.setTextColor( colors[0] );
      tft.print( oldPM2 );
      tft.setCursor( 0, 150 );
      tft.setTextColor( colors[colorCounter] );
      oldPM2 = "";
      oldPM2.concat( "PM2 " + String(px_eData.PM2) + "ug/m3" );
      tft.println( oldPM2 );
      tft.setCursor( 0, 180 );
      tft.setTextColor( colors[0] );
      tft.print( oldPower );
      tft.setCursor( 0, 180 );
      tft.setTextColor( colors[colorCounter] );
      oldPower = "";
      oldPower.concat(  String(px_eData.WSV) + " Volts" );
      //oldPower.concat(  String(px_eData.WSV) + "V " + String(int(px_eData.WSC * 1000.0f)) + "mA " + String((int(px_eData.WSP * 1000.0f))) + "mW" );
      tft.println( oldPower );
      colorCounter++;
      if ( colorCounter > (numOfColors - 1) )
      {
        colorCounter = 1;
      }
      Tick = !Tick;
      //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
    } //if ( xQueueReceive(xQ_eData, &px_eData, portMAX_DELAY) == pdTRUE )
  } //for (;;)
  vTaskDelete( NULL );
} //void fDoTheDisplayTHing( void * parameter )
////
void fmqttWatchDog( void * paramater )
{
  int UpdateImeTrigger = 86400; //seconds in a day
  int UpdateTimeInterval = 86300; // 1st time update in 100 counts
  int maxNonMQTTresponse = 5;
  for (;;)
  {
    vTaskDelay( 1000 );
    if ( mqttOK >= maxNonMQTTresponse )
    {
      ESP.restart();
    }
    xSemaphoreTake( sema_mqttOK, portMAX_DELAY );
    mqttOK++;
    xSemaphoreGive( sema_mqttOK );
    UpdateTimeInterval++; // trigger new time get
    if ( UpdateTimeInterval >= UpdateImeTrigger )
    {
      TimeSet = false; // sets doneTime to false to get an updated time after a days count of seconds
      UpdateTimeInterval = 0;
    }
  }
  vTaskDelete( NULL );
}
////
float fCalulate_IAQ_Index( int gasResistance, float Humidity)
{
  float hum_baseline = 40.0f;
  float hum_weighting = 0.25f;
  float gas_offset = 0.0f;
  float hum_offset = 0.0f;
  float hum_score = 0.0f;
  float gas_score = 0.0f;
  gas_offset = oGasResistanceBaseLine - float( gasResistance );
  hum_offset = float( Humidity ) - hum_baseline;
  // calculate hum_score as distance from hum_baseline
  if ( hum_offset > 0.0f )
  {
    hum_score = 100.0f - hum_baseline - hum_offset;
    hum_score /= ( 100.0f - hum_baseline );
    hum_score *= ( hum_weighting * 100.0f );
  } else {
    hum_score = hum_baseline + hum_offset;
    hum_score /= hum_baseline;
    hum_score *= ( 100.0f - (hum_weighting * 100.0f) );
  }
  //calculate gas score as distance from baseline
  if ( gas_offset > 0.0f )
  {
    gas_score = float( gasResistance ) / oGasResistanceBaseLine;
    gas_score *= ( 100.0f - (hum_weighting * 100.0f ) );
  } else {
    gas_score = 100.0f - ( hum_weighting * 100.0f );
  }
  return ( hum_score + gas_score );
} //void fCalulate_IAQ_Index( int gasResistance, float Humidity):
////
void fDoParticleDetector( void * parameter )
{
  /*
    ug/m3     AQI                 Lvl AQ (Air Quality)
    (air Quality Index)
    0-35     0-50                1   Excellent
    35-75    51-100              2   Average
    75-115   101-150             3   Light pollution
    115-150  151-200             4   moderate
    150-250  201-300             5   heavy
    250-500  >=300               6   serious
  */
  float ADbits = 4095.0f;
  float uPvolts = 3.3f;
  float adcValue = 0.0f;
  float dustDensity = 0.0f;
  float Voc = 0.6f; // Set the typical output voltage, when there is zero dust.
  const float K = 0.5f; // Use the typical sensitivity in units of V per 100ug/m3.
  xEventGroupWaitBits (eg, evtWaitForBME, pdTRUE, pdTRUE, portMAX_DELAY );
  TickType_t xLastWakeTime = xTaskGetTickCount();
  const TickType_t xFrequency = 100; //delay for mS
  for (;;)
  {
    //enable sensor led
    gpio_set_level( GPIO_NUM_4, HIGH ); // set gpio 4 to high to turn on sensor internal led for measurement
    esp_timer_start_once( oneshot_timer, 280 ); // trigger one shot timer for a 280uS timeout, warm up time.
    xEventGroupWaitBits (eg, evtDoParticleRead, pdTRUE, pdTRUE, portMAX_DELAY ); // event will be triggered by the timer expiring, wait here for the 280uS
    adcValue = float( adc1_get_raw(ADC1_CHANNEL_0) ); //take a raw ADC reading from the dust sensor
    gpio_set_level( GPIO_NUM_4, LOW );//Shut off the sensor LED
    adcValue = ( adcValue * uPvolts ) / ADbits; //calculate voltage
    dustDensity = (adcValue / K) * 100.0; //convert volts to dust density
    if ( dustDensity < 0.0f )
    {
      dustDensity = 0.00f; // make negative values a 0
    }
    if ( xSemaphoreTake( sema_PublishPM, 0 ) == pdTRUE )  // don't wait for semaphore to be available
    {
      xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY );
      //log_i( "ADC volts %f Dust Density = %ug / m3 ", adcValue, dustDensity ); // print the calculated voltage and dustdensity
      MQTTclient.publish( topicInsidePM, String(dustDensity).c_str() );
      xSemaphoreGive( sema_MQTT_KeepAlive );
      x_eData.PM2 = dustDensity;
    }
    xLastWakeTime = xTaskGetTickCount();
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
    //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
}// end fDoParticleDetector()
////
void DoTheBME680Thing( void *pvParameters )
{
  SPI.begin(); // initialize the SPI library
  vTaskDelay( 10 );
  if (!bme.begin()) {
    log_i("Could not find a valid BME680 sensor, check wiring!");
    while (1);
  }
  // Set up oversampling and filter initialization
  bme.setTemperatureOversampling(BME680_OS_8X);
  bme.setHumidityOversampling(BME680_OS_2X);
  bme.setPressureOversampling(BME680_OS_4X);
  bme.setIIRFilterSize(BME680_FILTER_SIZE_3);
  bme.setGasHeater(320, 150); // 320*C for 150 ms
  //wait for a mqtt connection
  while ( !MQTTclient.connected() )
  {
    vTaskDelay( 250 );
  }
  xEventGroupSetBits( eg, evtWaitForBME );
  TickType_t xLastWakeTime    = xTaskGetTickCount();
  const TickType_t xFrequency = 1000 * 15; //delay for mS
  String bmeInfo = "";
  bmeInfo.reserve( 100 );
  for (;;)
  {
    x_eData.Temperature  = bme.readTemperature();
    x_eData.Temperature  = ( x_eData.Temperature * 1.8f ) + 32.0f; // (Celsius x 1.8) + 32
    x_eData.Pressure     = bme.readPressure();
    x_eData.Pressure     = x_eData.Pressure / 133.3223684f; //converts to mmHg
    x_eData.Humidity     = bme.readHumidity();
    x_eData.IAQ          = fCalulate_IAQ_Index( bme.readGas(), x_eData.Humidity );
    //log_i( " temperature % f, Pressure % f, Humidity % f IAQ % f", x_eData.Temperature, x_eData.Pressure, x_eData.Humidity, x_eData.IAQ);
    bmeInfo.concat( String(x_eData.Temperature, 2) );
    bmeInfo.concat( "," );
    bmeInfo.concat( String(x_eData.Pressure, 2) );
    bmeInfo.concat( "," );
    bmeInfo.concat( String(x_eData.Humidity, 2) );
    bmeInfo.concat( "," );
    bmeInfo.concat( String(x_eData.IAQ, 2) );
    xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY );
    if ( MQTTclient.connected() )
    {
      MQTTclient.publish( topicInsideInfo, bmeInfo.c_str() );
    }
    xSemaphoreGive( sema_MQTT_KeepAlive );
    xSemaphoreGive( sema_PublishPM ); // release publish of dust density
    xSemaphoreTake( sema_mqttOK, portMAX_DELAY );
    mqttOK ++;
    xSemaphoreGive( sema_mqttOK );
    xQueueOverwrite( xQ_eData, (void *) &x_eData );// send data to display
    //
    bmeInfo = ""; // empty the string buffer
    findDewPointWithHumidity( x_eData.Humidity, x_eData.Temperature );
    xLastWakeTime = xTaskGetTickCount();
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
    // log_i( "DoTheBME280Thing high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete ( NULL );
}
////
/*
  Important to not set vTaskDelay/vTaskDelayUntil to less then 10. Errors begin to develop with the MQTT and network connection.
  makes the initial wifi/mqtt connection and works to keeps those connections open.
*/
void MQTTkeepalive( void *pvParameters )
{
  sema_MQTT_KeepAlive   = xSemaphoreCreateBinary();
  xSemaphoreGive( sema_MQTT_KeepAlive ); // found keep alive can mess with a publish, stop keep alive during publish
  MQTTclient.setKeepAlive( 90 ); // setting keep alive to 90 seconds makes for a very reliable connection, must be set before the 1st connection is made.
  TickType_t xLastWakeTime = xTaskGetTickCount();
  const TickType_t xFrequency = 250; //delay for ms
  for (;;)
  {
    //check for a is-connected and if the WiFi 'thinks' its connected, found checking on both is more realible than just a single check
    if ( (wifiClient.connected()) && (WiFi.status() == WL_CONNECTED) )
    {
      xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY ); // whiles MQTTlient.loop() is running no other mqtt operations should be in process
      MQTTclient.loop();
      xSemaphoreGive( sema_MQTT_KeepAlive );
    }
    else {
      log_i( "MQTT keep alive found MQTT status % s WiFi status % s", String(wifiClient.connected()), String(WiFi.status()) );
      if ( !(wifiClient.connected()) || !(WiFi.status() == WL_CONNECTED) )
      {
        connectToWiFi();
      }
      connectToMQTT();
    }
    //log_i( " high watermark % d",  uxTaskGetStackHighWaterMark( NULL ) );
    xLastWakeTime = xTaskGetTickCount();
    vTaskDelayUntil( &xLastWakeTime, xFrequency );
  }
  vTaskDelete ( NULL );
}
////
void connectToMQTT()
{
  byte mac[5]; // create client ID from mac address
  WiFi.macAddress(mac); // get mac address
  String clientID = String(mac[0]) + String(mac[4]) ; // use mac address to create clientID
  while ( !MQTTclient.connected() )
  {
    MQTTclient.connect( clientID.c_str(), mqtt_username, mqtt_password );
    vTaskDelay( 250 );
  }
  MQTTclient.setCallback( mqttCallback );
  MQTTclient.subscribe  ( topicOK );
  MQTTclient.subscribe  ( topicRemainingMoisture_0 );
  MQTTclient.subscribe  ( topicWindSpeed );
  MQTTclient.subscribe  ( topicWindDirection );
  MQTTclient.subscribe  ( topicRainfall );
  MQTTclient.subscribe  ( topicWSVolts );
  MQTTclient.subscribe  ( topicWSCurrent );
  MQTTclient.subscribe  ( topicWSPower );
  MQTTclient.subscribe  ( topicDPnWI );
} //void connectToMQTT()
void connectToWiFi()
{
  int TryCount = 0;
  while ( WiFi.status() != WL_CONNECTED )
  {
    TryCount++;
    WiFi.disconnect();
    WiFi.begin( SSID, PASSWORD );
    vTaskDelay( 4000 );
    if ( TryCount == 10 )
    {
      ESP.restart();
    }
  }
  WiFi.onEvent( WiFiEvent );
}
////
float findDewPointWithHumidity( float humi, float temperature )
{
  //Celcius
  float ans =  (temperature - (14.55 + 0.114 * temperature) * (1 - (0.01 * humi)) - pow(((2.5 + 0.007 * temperature) * (1 - (0.01 * humi))), 3) - (15.9 + 0.117 * temperature) * pow((1 - (0.01 * humi)), 14));
  //log_i( "%f", ans );
  return ans;
}
void loop() { }
////

1 Like

My inclination would be to treat current and historical data the same way and always add readings to local storage. Then, when there is connectivity, send data and flag it as sent. Normally of course, the length of time that data will be retained will be very short.

Thanks @Idahowalker.......the picture you posted earlier....do you have a higher resolution pic? Curious to get a clearer view of you ESP connections

Like I said I may be overthinking some of this - but if this works for one then expanding it will be basically copy/paste

The idea of using and RPi is good....but as I have some spares PCs, I am inclined to try and make sue of them first before investing in an RPi

But I think now I need some ESPs and do some testing....

Thanks all for the input

Have a look at the MySensors website. There's a lot of information on there about their system and how to build various remote sensor devices. Their network has the ability to pass data back via intermediate nodes so the actual sensor node doesn't have to be in range of the gateway node for their system to work. All their nodes are generally based around 328P micros (same as Arduino UNO, Nanoe etc).

Personally, I tried their system - initially with NRF24L01 radio modules but gave up and moved to RFM69 radio modules due to poor range. Unfortunately their software didn't work at all with my RFM69 modules - not sure why as they are supported. The RFM69 library from LowPower Lab worked a treat.

It didn't take long for me to put together some code for the sensor nodes and the gateway node (that's what MySensors calls the node linked to the PC) to appear to speak the MySensors serial protocol.

I'm primarily just doing data collection at the moment so each sensor node sleeps for around an hour using the built-in watchdog timer (which isn't particularly accurate - but it's good enough), and then transmits temperature (from a DS18B20) and battery voltage (potential divider).

It all gets routed via the gateway (another simple 328P board powered over the USB-Serial connector) to a thin client PC (a 10Zig 5848 that I picked up on ebay for around 20GBP). That runs Linux Lite and a program called Domoitcz.

My sensor boards are clones of the Low Power Lab Moteino board. The board also supports LoRa radio modules as well.

MQTT Broker can be ran on a PC.

Node-Red can also be ran on a PC and is a lot easier to use then writing your own Python Program.

Node_red is a way to make a user interface from the MQTT data. Oh, NodeRed does much much more. https://nodered.org/

But that is what everyone is trying to sell us. My wife was speaking to a telemarketer the other day and completely laughed them off when they mentioned that Microsoft was involved. :grin:

Worth the effort though.