Reading data using Processing

I've been tinkering with my arduino Wifi sketch (uno with esp8266) and a processing sketch.

With the processing.net library and the WiFiEsp library on the arduino side I can use client.print to send strings from the arduino to my PC and print them to the console.

What I want to do now is send other forms of data (like int returns from my sensors) and rather than print them to the console, have the data written somewhere where it can be used to draw on processing. Eventually I want to be sending data from multiple sensors that each can be visualised with a different graph or gauge, ideally in the same processing sketch.

Here is an example of my working code - I tried replacing my client.print statements in loop() with client.write, and using readBytes() on the processing side but no cigar so far. Any and all advice would be welcome at this point!

Thanks for reading.

#include "WiFiEsp.h"

// Emulate Serial1 on pins 6/7 if not present
#ifndef HAVE_HWSERIAL1
#include "SoftwareSerial.h"
SoftwareSerial Serial1(4,7); // RX, TX
#endif


int status = WL_IDLE_STATUS;     // the Wifi radio's status
const int fanPin = 11;
const int solenoidPin1 = 12;
const int solenoidPin2 = 13;
int solenoidState1 = HIGH;
int solenoidState2 = HIGH;
unsigned long previousMillis = 0;
const long runTime = 5000;
const long interval = 60000;
const int pressureSensor = A0;
int pressureVal;
int pressure;

void setup()
{
  // initialize serial for debugging
  Serial.begin(115200);
  // initialize serial for ESP module
  Serial1.begin(9600);
  // initialize ESP module
  WiFi.init(&Serial1);

  // check for the presence of the shield
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    // don't continue
    while (true);
  }

  // attempt to connect to WiFi network
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to WPA SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network
    status = WiFi.begin(ssid, pass);
  }

  Serial.println("You're connected to the network");
  pinMode(fanPin, OUTPUT);
  analogWrite(fanPin, 150);
}

void loop()
{
  // print the network connection information every 10 seconds
 const uint16_t port = 5204;
 const char * host = "192.168.0.26";
 Serial.print("Connecting to host. . .");

 WiFiEspClient client;
 
int pressureVal = analogRead(pressureSensor);
pressure = map(pressureVal, 175, 900, 0, 100);
  float bar = pressure;


 if(client.connect(host, port)) {
  Serial.println("Sending data");
  client.print(bar/10);
  client.print(" BAR");
 }
 
 client.print("Signal strength : ");
 client.print(WiFi.RSSI());
 Serial.println("Closing connection");
 client.stop();
  
  Serial.println();
  printCurrentNet();
  printWifiData();
  
  delay(5000);
}

void printWifiData()
{
  // print your WiFi shield's IP address
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print your MAC address
  byte mac[6];
  WiFi.macAddress(mac);
  char buf[20];
  sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X", mac[5], mac[4], mac[3], mac[2], mac[1], mac[0]);
  Serial.print("MAC address: ");
  Serial.println(buf);
}

void printCurrentNet()
{
  // print the SSID of the network you're attached to
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print the MAC address of the router you're attached to
  byte bssid[6];
  WiFi.BSSID(bssid);
  char buf[20];
  sprintf(buf, "%02X:%02X:%02X:%02X:%02X:%02X", bssid[5], bssid[4], bssid[3], bssid[2], bssid[1], bssid[0]);
  Serial.print("BSSID: ");
  Serial.println(buf);

  // print the received signal strength
  long rssi = WiFi.RSSI();
  Serial.print("Signal strength (RSSI): ");
  Serial.println(rssi);
}
import processing.net.*;
Server myServer;
int dataIn;

void setup() {
  size(2000, 800);
  background(255);
  // Starts a myServer on port 5204
  myServer = new Server(this, 5204); 
  
}

void draw() {
 // Get the next available client
  Client thisClient = myServer.available();
  // If the client is not null, and says something, display what it said
  if (thisClient !=null) {
    String whatClientSaid = thisClient.readString();
    if (whatClientSaid != null) {
       println(thisClient.ip() + "t" + whatClientSaid);
    }
  }
}

How about looking into using MQTT?

I use a RPi4 running a MQTT Broker service.

I have several ESP32's on a local network, that, wireless, publish and subscribe to the MQTT Broker. A Python script, running on the RPi4, subscribed to the MQTT Broker, reads incoming data from the EPS32's, and routes the data between the ESP32's and my web sites database. I can then use my web site to make adjustments to the ESP32's, such as changing the temperature of the HVAC or turn off/on the sunlamp providing the house plants with goodness.

Hi Idaho, thanks for the reply.

I'm having a look into it now, is there any particular broker you'd recommend? I'll be running it on my PC or possibly a NAS.

MQTT, https://mqtt.org/

Also, Node-Red is a great troubleshooting tool for MQTT. Many people just use Node-Red as their 'control' panel. Once Node-Red is started it requires a computer needs to stop the service, after its no longer required for troubleshooting.

Hi Idaho,

I've set up an MQTT broker now that is publishing from the arduino and I'm subscribed in node-red.

After much swearing I've managed to get node-red to receive my sensor doubles by converting them to strings. Is there a better way to do this? (seen a lot of posts saying that you shouldn't be using strings if at all possible).

Code snippet below

void sendPressureData() {
  pressureVal =  200;           //    analogRead(pressureSensor);
  pressureBar = mapf(pressureVal, 175, 900, 0.0, 10.00);
  dtostrf(pressureBar, 4, 2, pressureData);
  if (currentMillis - previousPressureMillis >= pressureDataDelay) {
    client.publish("pressureData", pressureData);
     
  previousPressureMillis = currentMillis;
  }
}

Using a 32bit uController do you require 64bit floats, doubles?

The MQTT payload is passed as a string. For floats I use MQTTclient.publish( topicInsideTemp, String(ePtr[0]).c_str() ); to send the float data and ePtr[6] = String(strPayloadPtr).toFloat(); to receive floats.

No, according to the arduino page though it shouldn't make a difference, it'll be 32-bit anyway.

I haven't really got a grasp of the pointers yet so will keep that one for later if I have any troubles, thanks for all your help!

trojanhawrs:
No, according to the arduino page though it shouldn't make a difference, it'll be 32-bit anyway.

I haven't really got a grasp of the pointers yet so will keep that one for later if I have any troubles, thanks for all your help!

The previous post was a demonstration of sending and receiving floats, of converting the floats to strings for publishing.

The code is from a project that is using a ESP32-WROVER-B with 8MB of PSRAM. I am using a pointer to access the extra ram where I have some large array's of data in storage.

What library are you using on the ESP side? I've got it publishing OK but the messaging is really unreliable on the subscribe side on my arduino

Here is my ESP-WROVER-B code in several parts

#include <WiFi.h>
#include <PubSubClient.h>
#include "certs.h"
#include "sdkconfig.h"
#include "esp_system.h"
#include "freertos/FreeRTOS.h"
#include "freertos/task.h"
#include "freertos/timers.h"
#include "freertos/event_groups.h"
#include <SPI.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_BME280.h>
#include <Adafruit_GFX.h>
#include <Adafruit_SSD1351.h>
#include "esp32-hal-psram.h"
#include "esp_himem.h"
#include "time.h"
//
//
hw_timer_t * timer = NULL;
EventGroupHandle_t eg;
#define evtDoDisplay ( 1 << 1 )
#define evtCollectHistory ( 1 << 2 )
#define evtParseMQTT ( 1 << 5 )
#define evtDoTheBME280Thing ( 1 << 8 )
#define evtDoTheHumidityThing ( 1 << 9 )
#define evtDoTheSunLampThing ( 1 << 10 )
#define evtMQTT_Disconnect ( 1 << 11 )
// the events in this event groups tasks have been given a delay to avoid a possible race condition.
#define evtGroupBits ( evtCollectHistory | evtDoTheBME280Thing | evtDoTheHumidityThing | evtDoTheSunLampThing | evtDoDisplay )
//
Adafruit_SSD1351 tft = Adafruit_SSD1351( 128, 128, GPIO_NUM_27, GPIO_NUM_12, GPIO_NUM_13, GPIO_NUM_14, GPIO_NUM_26 );
////
Adafruit_BME280 bme( GPIO_NUM_5 ); // hardware SPI
////
WiFiClient wifiClient;
PubSubClient MQTTclient( mqtt_server, mqtt_port, wifiClient );
////
////
// memory pointers for variables stored in himem, WROVER PSRAM.
/*
   ePtr = envirtonemtal pointers to float values
   [0] = inside temperature
   [1] = inside humidity
   {2} = inside pressure
   [3] = outside temperature
   [4] = outside humidity
   [5] = outside pressure
   [6] = outside AQ index
*/
float *ePtr;
float *oPressureHistoryPtr;
float *oIAQ_HistoryPtr;
float *oHumidityHistoryPtr;
float *oTemperaturePtr;
////
char *strPayloadPtr;
char *str_eTopicPtr;
////
////
int *ColorPalettePtr;
/*
   int data pointer, used to store globals
   IntDataPtr[0] int DataCells = 50
   IntDataPtr[1] int arrayCellPtr = 0;
   IntDataPtr[2] int humidity set point
   IntDataPtr[3] humidity control enable
   IntDataPtr[4] = sunlamp on
   IntDataPtr[5] = sun lamp enable
*/
int *IntDataPtr;
////
// RTC_DATA_ATTR int bootCount = 0; // assign a memory location in RTC FAST RAM, an experiment remnent
////
////
SemaphoreHandle_t sema_HistoryCompleted;
SemaphoreHandle_t sema_MQTT_Parser;;
SemaphoreHandle_t sema_TheHumidityThing;
SemaphoreHandle_t sema_DoTheSunLampTHing;
SemaphoreHandle_t sema_MQTT_KeepAlive;
////
volatile int iDoTheBME280Thing = 0;
////
void IRAM_ATTR onTimer()
{
  BaseType_t xHigherPriorityTaskWoken;
  iDoTheBME280Thing++;
  if ( iDoTheBME280Thing == 60000 )
  {
    xEventGroupSetBitsFromISR( eg, evtGroupBits, &xHigherPriorityTaskWoken );
    iDoTheBME280Thing = 0;
  }
}
////
void setup()
{
  //
  gpio_config_t io_cfg = {};
  io_cfg.mode = GPIO_MODE_OUTPUT;
  //bit mask of the pins to set
  io_cfg.pin_bit_mask = ( (1ULL << GPIO_NUM_0) | (1ULL << GPIO_NUM_15) | (1ULL << GPIO_NUM_2) );
  //configure GPIO with the given settings
  gpio_config(&io_cfg);
  REG_WRITE(GPIO_OUT_W1TS_REG, BIT0);
  REG_WRITE(GPIO_OUT_W1TC_REG, BIT12);
  REG_WRITE(GPIO_OUT_W1TC_REG, BIT15);
  //
  /* Use 4th hardware timer of 4.
    1 tick 1/(80MHZ/80) = 1us set divider 80 and count up.
    Attach onTimer function to timer
    Set alarm to call timer ISR, every 1000uS and repeat / reset ISR (true) after each alarm
    Start an timer alarm
  */
  timer = timerBegin( 3, 80, true );
  timerAttachInterrupt( timer, &onTimer, true );
  timerAlarmWrite(timer, 1000, true);
  timerAlarmEnable(timer);
  //
  // memory allocations for items to be stored in PSRAM
  log_i("before Free PSRAM: %d", ESP.getFreePsram());
  IntDataPtr = (int*)ps_calloc( 15, sizeof(int) );
  IntDataPtr[0] = 60; //const int DataCells = 60;
  oPressureHistoryPtr = (float*)ps_calloc( IntDataPtr[0], sizeof(float) );// oPressureHistoryPtr = (float*)ps_calloc( DataCells, sizeof(float) );
  oIAQ_HistoryPtr = (float*)ps_calloc( IntDataPtr[0], sizeof(float) );//oIAQ_HistoryPtr = (float*)ps_calloc( DataCells, sizeof(float) );
  oHumidityHistoryPtr = (float*)ps_calloc( IntDataPtr[0], sizeof(float) );
  oTemperaturePtr = (float*)ps_calloc( IntDataPtr[0], sizeof(float) );
  ePtr =  (float*)ps_calloc( 15, sizeof(float) );
  strPayloadPtr = (char *)ps_calloc(300, sizeof(char) );
  for ( int i = 0; i < 299; i++ )
  {
    strPayloadPtr[i] = '\0';
    strPayloadPtr[i] = '\0';
  }
  str_eTopicPtr = (char *)ps_calloc(300, sizeof(char) );
  ColorPalettePtr  = (int*)ps_calloc( 10, sizeof(int) );
  log_i("Total heap: %d", ESP.getHeapSize());
  log_i("Free heap: %d", ESP.getFreeHeap());
  log_i("Total PSRAM: %d", ESP.getPsramSize());
  log_i("Free PSRAM: %d", ESP.getFreePsram());
  // popuate color palette for display use
  ColorPalettePtr[0] = 0x0000; //BLACK
  ColorPalettePtr[1] = 0x001F; //BLUE
  ColorPalettePtr[2] = 0xF800; //RED
  ColorPalettePtr[3] = 0x07E0; //GREEN
  ColorPalettePtr[4] = 0x07FF; //CYAN
  ColorPalettePtr[5] = 0xF81F; //MAGENTA
  ColorPalettePtr[6] = 0xFFE0; //YELLOW
  ColorPalettePtr[7] = 0xFFFF; //WHITE
  ColorPalettePtr[8] = 0xFFFFD8; //LIGHTYELLOW
  ColorPalettePtr[9] = 0xFF8040; //BROWN
  //
  ePtr[2] = 50; // set a default humidity level
  ePtr[3] = 0; // set humidity control to off
  //
  eg = xEventGroupCreate();
  SPI.begin();
  bme.begin();
  while ( !bme.begin() )
  {
    log_i( "Waiting on response from BME280");
    vTaskDelay( 1000 );
  }
  tft.begin();
  tft.fillScreen(0x0000);
  //
  sema_MQTT_Parser = xSemaphoreCreateBinary();
  sema_HistoryCompleted = xSemaphoreCreateBinary();
  sema_TheHumidityThing = xSemaphoreCreateBinary();
  sema_DoTheSunLampTHing = xSemaphoreCreateBinary();
  sema_MQTT_KeepAlive = xSemaphoreCreateBinary();
  xSemaphoreGive( sema_DoTheSunLampTHing );
  xSemaphoreGive( sema_HistoryCompleted );
  xSemaphoreGive( sema_TheHumidityThing );
  xSemaphoreGive( sema_MQTT_KeepAlive ); // found keep alive can mess with a publish, stop keep alive during publish
  ////
  xTaskCreatePinnedToCore( fparseMQTT, "fparseMQTT", 7000, NULL, 5, NULL, 1 ); // assign all to core 1, WiFi in use.
  xTaskCreatePinnedToCore( fCollectHistory, "fCollectHistory", 10000, NULL, 4, NULL, 1 );
  xTaskCreatePinnedToCore( MQTTkeepalive, "MQTTkeepalive", 7000, NULL, 2, NULL, 1 ); //this task makes a WiFi and MQTT connection.
  xTaskCreatePinnedToCore( fUpdateDisplay, "fUpdateDisplay", 50000, NULL, 5, NULL, 1 );
  xTaskCreatePinnedToCore( DoTheBME280Thing, "DoTheBME280Thing", 20000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fDoTheHumidityThing, "fDoTheHumidityThing", 2000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fDoTheSunLampThing, "fDoTheSunLampThing", 2000, NULL, 3, NULL, 1 );
  xTaskCreatePinnedToCore( fMQTT_Disconnect, "fMQTT_Disconnect", 2000, NULL, 2, NULL, 1 );
  ////
} //setup()
////
////
// disconnect causes the MQTT data presistance to clear the presisted data. outside environmental data is updated once every 5 minutes.
void fMQTT_Disconnect( void * parameter )
{
  int i = 0;
  for (;;)
  {
    xEventGroupWaitBits (eg, evtMQTT_Disconnect, pdTRUE, pdTRUE, portMAX_DELAY );
    if( i >= 60 )
    {
    xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY ); //
    MQTTclient.disconnect();
    xSemaphoreGive( sema_MQTT_KeepAlive );
    i = 0;
    } else {
      i++;
    }
  }
  vTaskDelete( NULL );
}
////
void GetTheTime()
{
  char* ntpServer = "2.us.pool.ntp.org";
  int gmtOffset_sec = -(3600 * 7 );
  int daylightOffset_sec = 3600;
  configTime(gmtOffset_sec, daylightOffset_sec, ntpServer);
  printLocalTime();
}
////
// http://www.cplusplus.com/reference/ctime/strftime/
////
int getHour()
{
  struct tm timeinfo;
  getLocalTime(&timeinfo);
  char _time[ 5 ];
  strftime( _time, 80, "%T", &timeinfo );
  return String(_time).toInt();
}
////
void printLocalTime() {
  struct tm timeinfo;
  getLocalTime(&timeinfo);
  char _time[ 80 ];
  strftime( _time, 80, "%T", &timeinfo );
  log_i( "%s", _time);
  //  }
}
////
void fDoTheSunLampThing( void * parameter )
{
  //int x = gpio_get_level( GPIO_NUM_15 ); // reads gpio pin state returns an int
  // IntDataPtr[4] = sunlamp on manual mode, automatic mode off for manual mode to work
  // IntDataPtr[5] = sun lamp enable automatic mode
  bool AlreadyOn = false;
  for (;;)
  {
    xEventGroupWaitBits (eg, evtDoTheSunLampThing, pdTRUE, pdTRUE, portMAX_DELAY );
    vTaskDelay( 6 );
    xSemaphoreTake( sema_DoTheSunLampTHing, portMAX_DELAY );
    // sun lamp on/off auto disabled
    if ( (IntDataPtr[4] == 1) && (IntDataPtr[5] == 0)  )
    {
      if ( !AlreadyOn )
      {
        REG_WRITE(GPIO_OUT_W1TS_REG, BIT15);
        AlreadyOn = true;
      } else {
        REG_WRITE(GPIO_OUT_W1TC_REG, BIT15);
        AlreadyOn = false;
      }
    } else if ( (IntDataPtr[4] == 0) && (IntDataPtr[5] == 1) ) // light off auto enabled
    {
      int _hour = getHour();
      if ( (_hour >= 7) && (_hour <= 17) )
      {
        REG_WRITE(GPIO_OUT_W1TS_REG, BIT15);
        AlreadyOn = true;
      }
      if ( (_hour < 7) || (_hour > 16) )
      {
        REG_WRITE(GPIO_OUT_W1TC_REG, BIT15);
        AlreadyOn = false;
      }
    } else {
      REG_WRITE(GPIO_OUT_W1TC_REG, BIT15);
      AlreadyOn = false;
    }
    xSemaphoreGive( sema_DoTheSunLampTHing );
    //log_i( "fDoTheSunLampThing high watermark %d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
} //void fCollectHistory( void * parameter )
////
// send a signal out to relay to energize or de-energize humidifier based on set point
void fDoTheHumidityThing( void * parameter )
{
  for (;;)
  {
    xEventGroupWaitBits (eg, evtDoTheHumidityThing, pdTRUE, pdTRUE, portMAX_DELAY );
    vTaskDelay( 4 );
    xSemaphoreTake( sema_TheHumidityThing, portMAX_DELAY );
    if ( IntDataPtr[3] == 1 )
    {
      if ( IntDataPtr[2] + 1 < (int)ePtr[2] )
      {
        REG_WRITE(GPIO_OUT_W1TC_REG, BIT2);
      }
      if ( (IntDataPtr[2] - 1) > (int)ePtr[2] )
      {
        // energize humidifier
        REG_WRITE(GPIO_OUT_W1TS_REG, BIT2);
      }
    } else {
      //de energize humidifier
      REG_WRITE(GPIO_OUT_W1TC_REG, BIT2);
    }
    xSemaphoreGive( sema_TheHumidityThing );
    //log_i( "fDoTheHumidityThing high watermark %d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
} //void fCollectHistory( void * parameter )
////
/*
   Collect history information
   task triggered by hardware timer once a minute
   stores history into PSRAM
   Using a semaphore to protect the PSRAM from multiple tasks access the same PSRM locations
*/
void fCollectHistory( void * parameter )
{
  int StorageTriggerCount = 59;
  for (;;)
  {
    xEventGroupWaitBits (eg, evtCollectHistory, pdTRUE, pdTRUE, portMAX_DELAY );
    StorageTriggerCount++; //triggered by the timer isr once a minute count 60 minutes
    if ( StorageTriggerCount == 60 )
    {
      xSemaphoreTake( sema_HistoryCompleted, portMAX_DELAY );
      oPressureHistoryPtr[IntDataPtr[1]] = ePtr[5];
      //oIAQ_HistoryPtr[IntDataPtr[1]] = ePtr[6];
      //oHumidityHistoryPtr[IntDataPtr[1]] = ePtr[4];
      //oTemperaturePtr[IntDataPtr[1]] = ePtr[3];
      IntDataPtr[1]++;
      xSemaphoreGive( sema_HistoryCompleted );
      if ( IntDataPtr[1] == IntDataPtr[0] )
      {
        IntDataPtr[1] = 0;
      }
      StorageTriggerCount = 0;
    }
    // log_i( " high watermark %d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete( NULL );
} //void fCollectHistory( void * parameter )
////
// Using a semaphore to protect the PSRAM from multiple tasks access the same PSRM locations
////
void fUpdateDisplay( void * parameter )
{
  // Color definitions
  // http://www.barth-dev.de/online/rgb565-color-picker/
  /*
    ColorPalettePtr[0] = 0x0000; //BLACK
    ColorPalettePtr[1] = 0x001F; //BLUE
    ColorPalettePtr[2] = 0xF800; //RED
    ColorPalettePtr[3] = 0x07E0; //GREEN
    ColorPalettePtr[4] = 0x07FF; //CYAN
    ColorPalettePtr[5] = 0xF81F; //MAGENTA
    ColorPalettePtr[6] = 0xFFE0; //YELLOW
    ColorPalettePtr[7] = 0xFFFF; //WHITE
    ColorPalettePtr[8] = 0xFFFFD8; //LIGHTYELLOW
    ColorPalettePtr[9] = 0xFF8040; //BROWN
  */
  for (;;)
  {
    xEventGroupWaitBits (eg, evtDoDisplay, pdTRUE, pdTRUE, portMAX_DELAY ); //
    vTaskDelay( 10 );
    tft.fillScreen( ColorPalettePtr[0] );
    tft.setTextColor( ColorPalettePtr[7] );
    tft.setCursor( 0, 0 );
    tft.print( "Inside"  );
    tft.setTextColor( ColorPalettePtr[3] );
    tft.setCursor( 0, 15 );
    tft.print( "Temp: " + String(ePtr[0]) + "C " + String((ePtr[0] * 9 / 5) + 32) + "F"  );
    tft.setCursor( 0, 25 );
    tft.print( "Humidity " + String(ePtr[2]) + "%" );
    //
    tft.setTextColor( ColorPalettePtr[7] );
    tft.setCursor( 0,  40 );
    tft.print( "Outside" );
    tft.setTextColor( ColorPalettePtr[6] );
    tft.setCursor( 0, 55 );
    tft.print( "Temperature: " + String(ePtr[3]) + "F" );
    tft.setTextColor( ColorPalettePtr[2] );
    tft.setCursor( 0, 65 );
    tft.print( "Humidity " + String(ePtr[4]) + "%" );
    tft.setCursor( 0, 75 );
    tft.setTextColor( ColorPalettePtr[1] );
    tft.print( "Pres. " + String(ePtr[5]) + "mmHg" );
    tft.setCursor( 0, 86 );
    //set the color of the value to be displayed
    if ( ePtr[6] < 51.0f )
    {
      tft.setTextColor( ColorPalettePtr[3] );
    }
    if ( (ePtr[6] >= 50.0f) && (ePtr[6] <= 100.0f) )
    {
      tft.setTextColor( ColorPalettePtr[6] );
    }
    if ( (ePtr[6] >= 100.0f) && (ePtr[6] <= 150.0f) )
    {
      tft.setTextColor( ColorPalettePtr[9] );
    }
    if ( (ePtr[6] >= 150.0f) && (ePtr[6] <= 200.0f) )
    {
      tft.setTextColor( ColorPalettePtr[2] );
    }
    if ( (ePtr[6] >= 200.00f) && (ePtr[6] <= 300.0f) )
    {
      tft.setTextColor( ColorPalettePtr[5] );
    }
    if ( (ePtr[6] > 300.0f) )
    {
      tft.setTextColor( ColorPalettePtr[7] );
    }
    tft.print( "AQ Index " + String(ePtr[6]) );
    tft.setTextColor( ColorPalettePtr[1] ); //set graph line color
    int rowRef = 110;
    int hRef = int( oPressureHistoryPtr[0] );
    int nextPoint = 2;
    int nextCol = 0;
    xSemaphoreTake( sema_HistoryCompleted, portMAX_DELAY );
    for (int i = 0; i < IntDataPtr[0]; i++)
    {
      int hDisplacement = hRef - int( oPressureHistoryPtr[i] ); // cell height displacement from base line
      tft.setCursor( nextCol , (rowRef + hDisplacement) );
      tft.print( "_" );
      nextCol += nextPoint;
    }
    tft.setCursor( (IntDataPtr[1] * nextPoint), (rowRef + 3) );
    tft.print( "I" );
    xSemaphoreGive( sema_HistoryCompleted );
    //     log_i( "fUpdateDisplay MEMORY WATERMARK %d", uxTaskGetStackHighWaterMark(NULL) );
  }
  vTaskDelete( NULL );
} // void fUpdateDisplay( void * parameter )
////
void DoTheBME280Thing( void *pvParameters )
{
  for (;;)
  {
    xEventGroupWaitBits (eg, evtDoTheBME280Thing, pdTRUE, pdTRUE, portMAX_DELAY ); //
    vTaskDelay( 2 );
    if ( !isnan(bme.readTemperature()) )
    {
      ePtr[0] = bme.readTemperature();
      ePtr[1] = bme.readPressure() / 133.3223684f; // mmHg
      ePtr[2] = bme.readHumidity();
      if ( MQTTclient.connected() ) // broker online, then send stuff
      {
        xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY ); // do not send whiles keep alive is in progress and pause keep-alive during send
        MQTTclient.publish( topicInsideTemp, String(ePtr[0]).c_str() );
        vTaskDelay( 2 ); // gives the Raspberry Pi 4 time to receive the message and process
        MQTTclient.publish( topicInsideHumidity, String(ePtr[1]).c_str() );
        vTaskDelay( 2 ); // no delay and RPi is still processing previous message
        MQTTclient.publish( topicInsidePressure, String(ePtr[2]).c_str() );
        xSemaphoreGive( sema_MQTT_KeepAlive );
      }
    }
    //log_i( "DoTheBME280Thing high watermark %d",  uxTaskGetStackHighWaterMark( NULL ) );
  }
  vTaskDelete ( NULL );
}
////
/*
    Impostant to not set vtaskDelay to less then 10. Errors begin to develop with the MQTT and network connection.
    makes the initial wifi/mqtt connection and works to keeps those connections open.
*/
void MQTTkeepalive( void *pvParameters )
{
  // setting must be set before a mqtt connection is made
  MQTTclient.setKeepAlive( 90 ); // setting keep alive to 90 seconds
  for (;;)
  {
    if ( (wifiClient.connected()) && (WiFi.status() == WL_CONNECTED) )
    {
      xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY ); //
      MQTTclient.loop();
      xSemaphoreGive( sema_MQTT_KeepAlive );
    }
    else {
      log_i( "MQTT keep alive found MQTT status %s WiFi status %s", String(wifiClient.connected()), String(WiFi.status()) );
      if ( !(WiFi.status() == WL_CONNECTED) )
      {
        connectToWiFi();
      }
      connectToMQTT();
    }
    vTaskDelay( 4900 );
  }
  vTaskDelete ( NULL );
}
////
void connectToMQTT()
{
  log_i( "connect to mqtt" );
  while ( !MQTTclient.connected() )
  {
    MQTTclient.connect( clientID, mqtt_username, mqtt_password );
    log_i( "connecting to MQTT" );
    vTaskDelay( 250 );
  }
  log_i("MQTT Connected");
  MQTTclient.setCallback( mqttCallback );
  MQTTclient.subscribe( mqtt_topic );
}
//
void connectToWiFi()
{
  log_i( "connect to wifi" );
  while ( WiFi.status() != WL_CONNECTED )
  {
    WiFi.disconnect();
    WiFi.begin( SSID, PASSWORD );
    log_i(" waiting on wifi connection" );
    vTaskDelay( 4000 );
  }
  log_i( "Connected to WiFi" );
  WiFi.onEvent( WiFiEvent );
  GetTheTime();
}
////
void fparseMQTT( void *pvParameters )
{
  xSemaphoreGive ( sema_MQTT_Parser );
  for (;;)
  {
    xEventGroupWaitBits (eg, evtParseMQTT, pdTRUE, pdTRUE, portMAX_DELAY ); //
    xSemaphoreTake( sema_MQTT_Parser, portMAX_DELAY );
    if ( (String)str_eTopicPtr == topicOutsideTemperature )
    {
      ePtr[3] = String(strPayloadPtr).toFloat();
    }
    if ( (String)str_eTopicPtr == topicOutsideHumidity )
    {
      ePtr[4] = String(strPayloadPtr).toFloat();
    }
    if ( (String)str_eTopicPtr == topicAQIndex )
    {
      ePtr[6] = String(strPayloadPtr).toFloat();
    }
    if ( String(str_eTopicPtr) == topicStateLED )
    {
      if ( String(strPayloadPtr).toInt() )
      {
        REG_WRITE(GPIO_OUT_W1TC_REG, BIT0);
      } else {
        REG_WRITE(GPIO_OUT_W1TS_REG, BIT0);
      }
    }
    if ( String(str_eTopicPtr) == topicOutsidePressure ) // see note below
    {
      ePtr[5] = String(strPayloadPtr).toFloat();
       xEventGroupSetBits( eg, evtMQTT_Disconnect ); // trigger tasks
    }
    if ( (String)str_eTopicPtr == topic_hum_enable )
    {
      xSemaphoreTake( sema_TheHumidityThing, portMAX_DELAY );
      IntDataPtr[3] = String(strPayloadPtr).toInt();
      xSemaphoreGive( sema_TheHumidityThing );
    }
    if ( (String)str_eTopicPtr == topic_hum_setpoint )
    {
      xSemaphoreTake( sema_TheHumidityThing, portMAX_DELAY );
      IntDataPtr[2] = String(strPayloadPtr).toInt();
      xSemaphoreGive( sema_TheHumidityThing );
    }
    if ( (String)str_eTopicPtr == topic_SunLampOn )
    {
      xSemaphoreTake( sema_DoTheSunLampTHing, portMAX_DELAY );
      IntDataPtr[4] = String(strPayloadPtr).toInt();
      xSemaphoreGive( sema_DoTheSunLampTHing );
    }
    if ( (String)str_eTopicPtr == topic_SunLampEnable )
    {
      xSemaphoreTake( sema_DoTheSunLampTHing, portMAX_DELAY );
      IntDataPtr[5] = String(strPayloadPtr).toInt();
      xSemaphoreGive( sema_DoTheSunLampTHing );
    }
    // clear pointer locations
    for ( int i = 0; i < 21; i++ )
    {
      strPayloadPtr[i] = '\0';
      strPayloadPtr[i] = '\0';
    }
    xSemaphoreGive( sema_MQTT_Parser );
  }
} // void fparseMQTT( void *pvParameters )
/*
   frequent disconnects make for more realible data receiption when data changes
   Pressure is the last data item sents, mqttkeepalive will cause reconnection
*/
////
// Important to get as much code out of the callback for realible operations.
////
static void mqttCallback(char* topic, byte * payload, unsigned int length)
{
  xSemaphoreTake( sema_MQTT_Parser, portMAX_DELAY);
  str_eTopicPtr = topic + '\0';
  int i = 0;
  for ( i; i < length; i++) {
    strPayloadPtr[i] = ((char)payload[i]);
  }
  strPayloadPtr[i] = '\0';
  xSemaphoreGive ( sema_MQTT_Parser );
  xEventGroupSetBits( eg, evtParseMQTT ); // trigger tasks
} // void mqttCallback(char* topic, byte* payload, unsigned int length)
////
// great trouble shooting tool when uncommented
////
void WiFiEvent(WiFiEvent_t event)
{
  // log_i( "[WiFi-event] event: %d\n", event );
  switch (event) {
    //    case SYSTEM_EVENT_WIFI_READY:
    //      log_i("WiFi interface ready");
    //      break;
    //    case SYSTEM_EVENT_SCAN_DONE:
    //      log_i("Completed scan for access points");
    //      break;
    //    case SYSTEM_EVENT_STA_START:
    //      log_i("WiFi client started");
    //      break;
    //    case SYSTEM_EVENT_STA_STOP:
    //      log_i("WiFi clients stopped");
    //      break;
    case SYSTEM_EVENT_STA_CONNECTED:
      log_i("Connected to access point");
      break;
    case SYSTEM_EVENT_STA_DISCONNECTED:
      log_i("Disconnected from WiFi access point");
      break;
    //    case SYSTEM_EVENT_STA_AUTHMODE_CHANGE:
    //      log_i("Authentication mode of access point has changed");
    //      break;
    //    case SYSTEM_EVENT_STA_GOT_IP:
    //      log_i ("Obtained IP address: %s",  WiFi.localIP() );
    //      break;
    //    case SYSTEM_EVENT_STA_LOST_IP:
    //      log_i("Lost IP address and IP address is reset to 0");
    //      //      vTaskDelay( 5000 );
    //      //      ESP.restart();
    //      break;
    //    case SYSTEM_EVENT_STA_WPS_ER_SUCCESS:
    //      log_i("WiFi Protected Setup (WPS): succeeded in enrollee mode");
    //      break;
    //    case SYSTEM_EVENT_STA_WPS_ER_FAILED:
    //      log_i("WiFi Protected Setup (WPS): failed in enrollee mode");
    //      //      ESP.restart();
    //      break;
    //    case SYSTEM_EVENT_STA_WPS_ER_TIMEOUT:
    //      log_i("WiFi Protected Setup (WPS): timeout in enrollee mode");
    //      break;
    //    case SYSTEM_EVENT_STA_WPS_ER_PIN:
    //      log_i("WiFi Protected Setup (WPS): pin code in enrollee mode");
    //      break;
    //    case SYSTEM_EVENT_AP_START:
    //      log_i("WiFi access point started");
    //      break;
    //    case SYSTEM_EVENT_AP_STOP:
    //      log_i("WiFi access point  stopped");
    //      //      WiFi.mode(WIFI_OFF);
    //      //      esp_sleep_enable_timer_wakeup( 1000000 * 2 ); // 1 second times how many seconds wanted
    //      //      esp_deep_sleep_start();
    //      break;
    //    case SYSTEM_EVENT_AP_STACONNECTED:
    //      log_i("Client connected");
    //      break;
    case SYSTEM_EVENT_AP_STADISCONNECTED:
      log_i("WiFi client disconnected");
    //      break;
    //    case SYSTEM_EVENT_AP_STAIPASSIGNED:
    //      log_i("Assigned IP address to client");
    //      break;
    //    case SYSTEM_EVENT_AP_PROBEREQRECVED:
    //      log_i("Received probe request");
    //      break;
    //    case SYSTEM_EVENT_GOT_IP6:
    //      log_i("IPv6 is preferred");
    //      break;
    //    case SYSTEM_EVENT_ETH_GOT_IP:
    //      log_i("Obtained IP address");
    //      break;
    default: break;
  }
}
////
void loop() { }

The code makes extensive use of the ESP32 API, which will not be applicable to your use. The task function fMQTTkeepAlive(), which might interest you, is what I use to make the initial connection, keeps the connection alive, and reconnects when I disconnect.

I do periodic disconnects to remove some persistent data on the MQTT Broker.

I have read that the setting MQTTclient.setKeepAlive( 90 ); is irrelevant but when I added the setting to my code the MQTTclient connection greatly improved its sustainability.

Because the code is multitasking, the use of a Semaphore, in this case,

 xSemaphoreTake( sema_MQTT_KeepAlive, portMAX_DELAY ); //
      MQTTclient.loop();
      xSemaphoreGive( sema_MQTT_KeepAlive );

,
is needed to prevent publishing of BME data and a keep.alive() from running at the 'same' time. A same time of data publishing and a keepalive() will mess with the publish.

You may find that it is important to keep as much code out of the MQTT callback as possible.
In regards to this code

for ( i; i < length; i++) {
    strPayloadPtr[i] = ((char)payload[i]);
  }
  strPayloadPtr[i] = '\0';

Whiles strPayloadPtr = payload; works and is much faster, I found the direct equals could cause issues, from time to time, with data reception.

Be aware of static void mqttCallback(char* topic, byte * payload, unsigned int length),
and that I am declaring the MQTT callback as static. I find by declaring that function as static causes the function to load faster. The static setting may cause you an issue or not.

Oh, if you are publishing multiple topics in a row, give the MQTT Broker, which is slow, time to receive and place the published info. I think I can use delays of less then a millisecond but have not tried it; no need.


The above posted code, recently, passed a 100 hour burn-in without producing an error or missing a data publish or subscribe.

Bloody hell, thats a lot of code! Most of it is way over my head I'm afraid.

I've tried setting the callback to static, no change. I also tried implementing your strpointer, badly apparently as I wasn't getting a proper output, but the output was still intermittent.

Should I try the keepalive function in my setup? Here's my code

#include <SPI.h>
#include "WiFiEsp.h"
#include "SoftwareSerial.h"
#ifndef HAVE_HWSERIAL1
#include <PubSubClient.h>
#include "IPAddress.h"
SoftwareSerial Serial1(4,7); // RX, TX
#endif

// Update these with values suitable for your hardware/network
IPAddress server(192, 168, 0, 26);
char ssid[] = "";            // your network SSID (name)
char pass[] = "";        // your network password
int status = WL_IDLE_STATUS;     // the Wifi radio's status
const int fanPin = 11;



WiFiEspClient WiFiclient;
PubSubClient client(WiFiclient);

long lastReconnectAttempt = 0;
long lastClientLoop = 0;

char pressureData[10];
int pressureVal;
double pressureBar;
const int pressureSensor = A0;
unsigned long previousPressureMillis = 0;
unsigned long currentMillis = 0;
const int pressureDataDelay = 5000;
unsigned long interval = 0;


void setup()
{
   // initialize serial for debugging
  Serial.begin(115200);
  // initialize serial for ESP module
  Serial1.begin(9600);
  // initialize ESP module
  WiFi.init(&Serial1);


  // check for the presence of the shield
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    // don't continue
    while (true);
  }

  // attempt to connect to WiFi network
  while ( status != WL_CONNECTED) {
    Serial.print("Attempting to connect to WPA SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network
    status = WiFi.begin(ssid, pass);
  }

  Serial.println("You're connected to the network");
  pinMode(fanPin, OUTPUT);
  analogWrite(fanPin, 150);
  client.setServer(server, 1883);
  client.setCallback(callback);
client.subscribe("inputs", 1);
client.subscribe("interval");
  delay(1500);
  lastReconnectAttempt = 0;
}


void loop()
{

  currentMillis = millis();
  if (!client.connected()) {
    long now = millis();
    if (now - lastReconnectAttempt > 5000) {
      lastReconnectAttempt = now;
      // Attempt to reconnect
      if (reconnect()) {
        lastReconnectAttempt = 0;
      }
    }
  } else {
    // Client connected
    client.loop();
  }

  sendPressureData();

}



static void callback(char* topic, byte* payload, unsigned int length) {

/* if (topic == "interval") {
  char buffer [256];
  interval = strtoul(buffer, 256, payload);
  client.publish("interval", "Interval updated");
  Serial.print("Current Interval Value: ");
  Serial.println(interval);
} */
  Serial.print("Message arrived in topic: ");
  Serial.println(topic);
 
  Serial.print("Message:");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  
   
 
  Serial.println();
  Serial.println("-----------------------");
  // handle message arrived
}

boolean reconnect() {

  if (client.connect("arduinoClient")) {
    // Once connected, publish an announcement...
    client.publish("outputs","hello world");
    // ... and resubscribe
    client.subscribe("inputs");
  }
  return client.connected();
}


double mapf(double x, double in_min, double in_max, double out_min, double out_max)
{
  return (x - in_min) * (out_max - out_min) / (in_max - in_min) + out_min;
}


void sendPressureData() {
  pressureVal =  848;           //    analogRead(pressureSensor);
  pressureBar = mapf(pressureVal, 175, 900, 0.0, 10.00);
  dtostrf(pressureBar, 4, 2, pressureData);
  if (currentMillis - previousPressureMillis >= pressureDataDelay) {
    client.publish("pressureData", pressureData);
     
  previousPressureMillis = currentMillis;
  }
}

E2A : I wonder if software serial is causing some kind of a problem, don't know that there is a solution to that if it is the case

I'm running the mosquitto broker on my PC FYI, assume that will be quicker than the PI

Whiles I review your code, your ESP8266 does not have the same capability of using the pointers in the same way I do.

Just below the Arduino IDE icon located to the right side, kinda under the x, to open the Monitor and under that is another icon. Click that icon, select new tab, name the tab certs.h, OK,go to the new tab and enter your secret info there.

Example.

const char* SSID = "SomeSSID";
const char* PASSWORD = "somepassword";

Save.

On the main tab, add #include "certs.h"

Now you can use SSID for your ssid and PASSWORD for the password, select your code, and paste it elsewhere, without having to deal with making sure your code is clean before posting it.

And, could you state the issue you are having and the possible location in code you think the issue stems from?

You are running a MQTT keepalive, client.loop(). Perhaps adding client.setKeepAlive( 90 ); could help?

Nice one, cheers for that. If I save it to my libraries will I be able to use that with every new sketch?

I honestly don't know if it's a code issue or not - I do receive messages sometimes but it's few and far between. I set up a little text publish node so that I can publish to "inputs" repeatedly and check the serial monitor, only 1/5 make it through. I also tried taking the serial print commands out and publishing a new packet to "outputs" when I get a callback but getting the same intermittent response. I'm fairly certain its not a broker problem, node has no problems receiving subscription messages.

I'll drop the keepalive in my setup and get back to you!

Re: the keepalive, didn't do anything noticeable.

I changed my callback function to the publish_in_callback examples.

// Callback function
void callback(char* topic, byte* payload, unsigned int length) {
  // In order to republish this payload, a copy must be made
  // as the orignal payload buffer will be overwritten whilst
  // constructing the PUBLISH packet.

  // Allocate the correct amount of memory for the payload copy
  byte* p = (byte*)malloc(length);
  // Copy the payload to the new buffer
  memcpy(p,payload,length);
  client.publish("outputs", p, length);
  // Free the memory
  free(p);
}

Interestingly that seems to be a little bit better, but theres still times that it won't register until the 3rd or 4th send.
Just seems like the arduino has selective hearing

client.subscribe("interval");
  delay(1500);
  lastReconnectAttempt = 0;

I'd remove the delay, as soon as the subscribes are completed they are active but the delay puts a stop to the thing do's.

What's the Arduino Uno do?

Running the ESP8266 without software serial would be a good test. See if the softwareserial is causing issues.

The delay is just in the setup, only called once. I did remove it, no change.

I'm literally just using the ESP8266 for WiFi capability, I'm using the Uno's IO's to talk to sensors and actuators. I know these ESP chips are very capable and there's some boards based around them with more pins but I already had an uno so, seemed the best option at the time.

I can probably make a workaround where I loop the publish on the node side until I get a response, it doesn't seem to like getting spammed though - its dropped the connection a few times.