ESP8266 MQTT publish issues: message seem to buffering before being sent.

Hi everyone,

for my first arduino project I am trying to acquire sensor data via WIFI using the MQTT protocol. At the moment I am still in my testing phase, running the following hardware:

Microcontroller: ESP8266 (Wemos D1 v3 mini)
MQTT broker: Mosquitto, running on my local machine for now (goal is to migrate to Raspberry Pi in a later stage)
'MQTT Client': Paho MQTT utility for testing purposes

I've created a basic code to check the MQTT functionality of my setup before actually connecting any inputs/ouputs to it. In the code below I try to publish a message to the broker every 1s. Using the Paho MQTT utility I connect to the broker to see how the messages from my ESP8266 are arriving to the broker.

For the first minute or so 1 message/second is arriving at the broker as I expect. However, after a while it seems my messages are no longer arriving every second anymore. The broker is only receiving messages every 10s or so after which it then receives all the missed messages in 1 go (see screenshot attached to this post). It looks as if the messages are somehow being buffered in the ESP and then sent out in a bundle. I think the issue is coming from the ESP side of my 'MQTT test setup' but I'm not 100% sure as I do not have a lot of experience on the topic. Does anyone have any experience with this behavior or an idea where this might be coming from?

Thanks in advance!!!

// Resource: https://techtutorialsx.com/2017/04/09/esp8266-connecting-to-mqtt-broker/

#include <ESP8266WiFi.h>
#include <PubSubClient.h>

//  network credentials
const char* ssid = "XXXX";
const char* password = "XXXX";
const char* mqttServer = "XXX.YYY.ZZZ.Z"; //
const int mqttPort = 1883; // standard Mosquitto port
/*const char* mqttUser = "YourMqttUser";
const char* mqttPassword = "YourMqttUserPassword";*/

// Declare Wifi client to allow connecting to specific IP and port. Declare PubSubClient for MQTT
WiFiClient wifiClient;
PubSubClient client(mqttServer, mqttPort, wifiClient);

// Setup timers
unsigned long DELAY_TIME = 1000; // 1 sec,
unsigned long delayStart = 0; // the time the delay started
bool delayRunning = false; // true if still waiting for delay to finish

// Variables
unsigned long count1 = 0;
char msg1[50];

///// Callback function to determine what happens to MQTT messages received///////////////////////
void callback(char* topic, byte* payload, unsigned int length) {
  String topicStr = topic; 
  Serial.print("Message arrived in topic: ");
  Serial.println(topicStr);
 
  Serial.print("Message:");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
  Serial.println("-----------------------");
}

///// Reconnect to network / MQTT in case connection is lost ////////////////////////////////////
void reconnect() {

  //attempt to connect to the wifi if connection is lost
  if(WiFi.status() != WL_CONNECTED){
    //debug printing
    Serial.print("Connecting to ");
    Serial.println(ssid);

    //loop while we wait for connection
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
    }

    //print out some more debug once connected
    Serial.println("");
    Serial.println("WiFi connected");  
    Serial.println("IP address: ");
    Serial.println(WiFi.localIP());
  }

  //make sure we are connected to WIFI before attemping to reconnect to MQTT
  if(WiFi.status() == WL_CONNECTED){
  // Loop until we're reconnected to the MQTT server
    while (!client.connected()) {
      Serial.print("Attempting MQTT connection...");

      // Generate client name based on MAC address and last 8 bits of microsecond counter
      String clientName;
      clientName += "esp8266-";
      uint8_t mac[6];
      WiFi.macAddress(mac);
      clientName += macToStr(mac);

      //if connected, subscribe to the topic(s) we want to be notified about
      //EJ: Delete "mqtt_username", and "mqtt_password" here if you are not using any 
      if (client.connect((char*) clientName.c_str() )) {  //EJ: Update accordingly with your MQTT account 
        Serial.print("\tMQTT Connected");
        // Test message in esp/test topic
        client.publish("esp/topic", "Hello from esp");
        // Subscribe to same topicstrstr to be able to receive messages
        client.subscribe("esp/topic");
      }
    }
  }
}

/////generate unique name from MAC addr//////////////////////////////////////////////////////////
String macToStr(const uint8_t* mac){

  String result;

  for (int i = 0; i < 6; ++i) {
    result += String(mac[i], 16);

    if (i < 5){
      result += ':';
    }
  }

  return result;
}

void setup(){
  Serial.begin(57600); delay(10);
// Handling function specification when MQTT message is received
  client.setCallback(callback);
  client.setKeepAlive(20000); // in ms, keep alive to make sure broker does not disconnect from ESP8266 client, 2x max time between messages

//start wifi subsystem
  WiFi.begin(ssid, password);
//attempt to connect to the WIFI network and then connect to the MQTT server
  reconnect();

//start timers
  delayStart = millis();
  delayRunning = true;
}

void loop(){
  if ((delayRunning && ((millis() - delayStart) >= DELAY_TIME))) {
    if(!client.connected()){
      reconnect();
    }
    delayStart += DELAY_TIME;
    count1 +=1;
    snprintf(msg1,75,"%ld", count1);
    client.publish("esp/test1", msg1);
    delay(200);
    Serial.println(msg1);
  }
  client.loop();
}

The first thing I would try is to remove the delay() inside loop(). Using delay() is generally a bad idea but especially when using communication stacks.

Another tip for future posts. Have a look at the Arduino WiFi examples. They use a header file called arduino_secrets.h with two definitions for the SSID and password. This allows you to post your code without the need to remove your secret data and replace them with XXX. And other user can just use their header file and can test the code if they want to.

That seemed to do the trick! Feeling a bit silly now, thanks!!!