Simultaneous MQTT and HTTP POST

Hi, is it possible to subscribe to a MQTT server -AND- http POST from the same Arduino?

Background: I'm running the home automation platform 'Home Assistant' (HA) and MQTT Server (Mosquito) on a RPi2, a HA timer is set to publish on a certain MQTT topic. I want the Arduino to subscribe to that topic and POST lines of HTTP code determined by the MQTT payload. The ultimate goal is a timer (alarm) on HA that will turn on/off the Denon amplifier ZONE 2.

Progress: The MQTT subscribe works and I can call void Z2_ON and void Z2_OFF, BUT the Arduino will not connect to the Denon. The POST code works fine, if I alter the order the sketch will POST and turn the Denon on, but will not connect to the MQTT server afterwards. It seems the Arduino can only make one connection.

More background: Home Assistant has a Denon module that will directly control the main Zone, but not Zone 2. I'm also looking at an Infra-red option, but would like to get this sketch operational. If anyone is familiar with HA and can recommend a better way to connect to the Arduino or POST HTTP I'd love to hear it.

Thanks in advance for any advice

/*
 Denon Remote Control Via MQTT using "POST" Command

Any string published on "arduino/remote" will be read into the string
"command" sent to the serial printer.  It will also be used to call relevant 
modules.

All code borrowed from folk a lot brighter than me.
 
*/

#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
byte mac[]    = {  0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
IPAddress ip(10, 16, 15, 131);          //Address of Arduino
IPAddress MQTTserver(10, 16, 15, 161);  //Running on the RPi2
IPAddress denon(10,16,15,180);          //Denon AVR-X3000
EthernetClient ethClient;
PubSubClient client(ethClient);
String command = "";


void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message arrived [");
  Serial.print(topic);
  Serial.print("] ");
  for (int i=0;i<length;i++) {   //Read MQTT payload into "command"
     command += ((char)payload[i]);
  }
  Serial.println(command);  //For testing only
  
}



void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    if (client.connect("arduino/remote")) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      //client.publish("arduino/remote","IR Romote Online"); //not needed
      // ... and resubscribe
      client.subscribe("arduino/remote");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void Z2_ON()
{

  if (ethClient.connect(denon, 80)) {
    Serial.println("connected");
    // Make a HTTP request:
    ethClient.println("POST /MainZone/index.put.asp HTTP/1.1");
    ethClient.println("Host: 10.16.15.180");
    ethClient.println("User-Agent: User-Agent: Arduino/1.0");
    ethClient.println("Accept: */*");
    ethClient.println("Accept-Language: en-us,en;q=0.5");
    ethClient.println("Accept-Encoding: gzip, deflate");
    ethClient.println("Connection: keep-alive");
    ethClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
    ethClient.println("X-Requested-With: XMLHttpRequest");
    ethClient.println("Referer: http://10.16.15.180/Zone2/index.html");
    ethClient.println("Content-Length: 74");
    ethClient.println("Cookie: ZoneName=ZONE2");
    ethClient.println("Pragma: no-cache");
    ethClient.println("");
    ethClient.println("cmd0=PutZone_OnOff%2FON&cmd1=aspMainZone_WebUpdateStatus%2F&ZoneName=ZONE2HTTP/1.0 200 OK");
    ethClient.println("Cache-Control: no-cache");
    ethClient.println("Connection: close");
    ethClient.println();
  } 
  else {
    // if you didn't get a connection to the server:
    Serial.println("connection failed");
  }


  
}

void Z2_OFF() {
  //No HTTP POST here as yet.
  Serial.println("Turn OFF Zone Two");
}

void setup()
{
  Serial.begin(9600);

  client.setServer(MQTTserver, 1883);
  client.setCallback(callback);

  Ethernet.begin(mac, ip);
  // Allow the hardware to sort itself out
  delay(1500);

}

void monitor_MQTT()
{
  if (!client.connected()) {
    reconnect();
  }
  client.loop();  
}

void loop()
{
  if (command == ""){
    monitor_MQTT();
  }
  else {
    Serial.println("MQTT Pub is " + command); //For Testing
    if (command == "Z2_ON"){
      Z2_ON();
    }
    if (command == "Z2_OFF"){
      Z2_OFF();
    }
    command = "";  //Reset String command and restart loop
  }

}

How much memory is your sketch using? Rather, misusing.

Serial.print(F("Message arrived ["));
Use the F() macro to keep string literals from being copied to SRAM at run time.

String command = "";

Stop using Strings, to keep from Swiss-cheesing the SRAM that you have.

There is no excuse for using Strings when you KNOW how many characters the string that the String instance is going to wrap will contain.

Prickles:

    ethClient.println("POST /MainZone/index.put.asp HTTP/1.1");

ethClient.println("Host: 10.16.15.180");
   ethClient.println("User-Agent: User-Agent: Arduino/1.0");
   ethClient.println("Accept: /");
   ethClient.println("Accept-Language: en-us,en;q=0.5");
   ethClient.println("Accept-Encoding: gzip, deflate");
   ethClient.println("Connection: keep-alive");
   ethClient.println("Content-Type: application/x-www-form-urlencoded; charset=UTF-8");
   ethClient.println("X-Requested-With: XMLHttpRequest");
   ethClient.println("Referer: http://10.16.15.180/Zone2/index.html");
   ethClient.println("Content-Length: 74");
   ethClient.println("Cookie: ZoneName=ZONE2");
   ethClient.println("Pragma: no-cache");
   ethClient.println("");
   ethClient.println("cmd0=PutZone_OnOff%2FON&cmd1=aspMainZone_WebUpdateStatus%2F&ZoneName=ZONE2HTTP/1.0 200 OK");
   ethClient.println("Cache-Control: no-cache");
   ethClient.println("Connection: close");
   ethClient.println();

Lots of bad stuff here.

  • You probably want HTTP/1.0 instead of HTTP/1.1 in the first line.
  • Sending a Host: header with an IP address is silly; it's supposed to be a hostname. Granted, if you're using HTTP/1.1 it must be present.
  • User-Agent, Accept, Accept-Language: you don't need these. The strings themselves are just eating memory on your Arduino.
  • Accept-Encoding gzip? You're saying you can handle compressed responses -- and you can't. Remove it.
  • Connection: You don't want keep-alive. We're not going to be sending requests in rapid succession where we need the performance of an always-open connection, and an always-open connection tends to clog things up anyway.
  • Content-type: not required, again uses memory.
  • X-Requested-With, Referer: not needed.
  • Content-Length: This is actually correct -- 74 characters long.
  • Cookie: Strange to be using this since it duplicates data in the POSTed message, but ok.
  • Pragma no-cache and Cache-Control: Unless you have transparent proxies in your home network (highly doubt that) these are unnecessary.
  • Cache-Control and Connection headers come late; you can't send more headers after you've sent the "blank line".

Finally your POSTed data has a "HTTP/1.0 200 OK" string on the end of it that should not be there.

PaulS:
How much memory is your sketch using? Rather, misusing.

Hi PaulS, thanks for your response, I appreciate your time.
Thanks for drawing my attention to the F() macro, I've had a quick read on the forum, I'll modify my code and get my head around the macro.

Re you question about usage, this is the message I get on upload “Sketch uses 16486 bytes (57%) of program storage space. Maximum is 28672 bytes. Global variables use 1313 bytes (51%) of dynamic memory, leaving 1247 bytes for local variables. Maximum is 2560 bytes.”
While it's not good code I don't think it's memory that is preventing to connection to the Denon.

Chagrin:
Lots of bad stuff here.

Hi Chagrin, likewise, thank you for taking the time out to look over my code and post your feedback, much appreciated.

You've probably guessed I'm not a HTTP guru, I used wireshark to capture the POST code from a Windows PC and simply pasted one line at a time until it turned on the Denon, which only happened after I posted the last line. I've used your feedback to prune all the superfluous code out and now have it to the bare minimum and it works when I run it in a separate sketch.

My problem is I still can't make a connection to the Denon if I have a connection to the MQTT server. The serial response is...

“Message arrived [arduino/remote] Z2_ON
MQTT Pub is Z2_ON
connection failed”

Modified Code is Here.....

/*
 Denon Remote Control Via MQTT using "POST" Command

Any string published on "arduino/remote" will be read into the string
"command" sent to the serial printer.  It will also be used to call relevant 
modules.

All code borrowed from folk a lot brighter than me.
 
*/

#include <SPI.h>
#include <Ethernet.h>
#include <PubSubClient.h>

// Update these with values suitable for your network.
byte mac[]    = {  0xDE, 0xED, 0xBA, 0xFE, 0xFE, 0xED };
IPAddress ip(10, 16, 15, 131);          //Address of Arduino
IPAddress MQTTserver(10, 16, 15, 161);  //Running on the RPi2
IPAddress denon(10,16,15,180);          //Denon AVR-X3000
EthernetClient ethClient;
PubSubClient client(ethClient);
String command = "";


void callback(char* topic, byte* payload, unsigned int length) {
  Serial.print(F("Message arrived ["));
  Serial.print(topic);
  Serial.print(F("] "));
  for (int i=0;i<length;i++) {   //Read MQTT payload into "command"
     command += ((char)payload[i]);
  }
  Serial.println(command);  //For testing only
  
}



void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print(F("Attempting MQTT connection..."));
    // Attempt to connect
    if (client.connect("arduino/remote")) {
      Serial.println("connected");
      // Once connected, publish an announcement...
      //client.publish("arduino/remote","IR Romote Online"); //not needed
      // ... and resubscribe
      client.subscribe("arduino/remote");
    } else {
      Serial.print(F("failed, rc="));
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      // Wait 5 seconds before retrying
      delay(5000);
    }
  }
}

void Z2_ON()
{

  if (ethClient.connect(denon, 80)) {
    Serial.println(F("connected"));
    // Make a HTTP request:
    ethClient.println("POST /MainZone/index.put.asp HTTP/1.1");
    ethClient.println("Host: 10.16.15.180");
    ethClient.println("Content-Length: 74");
    ethClient.println("");
    ethClient.println("cmd0=PutZone_OnOff%2FON&cmd1=aspMainZone_WebUpdateStatus%2F&ZoneName=ZONE2");
    ethClient.println("Connection: close");
    ethClient.println();
  } 
  else {
    // if you didn't get a connection to the server:
    Serial.println(F("connection failed"));
  }


  
}

void Z2_OFF() {
  //No HTTP POST here as yet.
  Serial.println(F("Turn OFF Zone Two"));
}

void setup()
{
  Serial.begin(9600);

  client.setServer(MQTTserver, 1883);
  client.setCallback(callback);

  Ethernet.begin(mac, ip);
  // Allow the hardware to sort itself out
  delay(1500);

}

void monitor_MQTT()
{
  if (!client.connected()) {
    reconnect();
  }
  client.loop();  
}

void loop()
{
  if (command == ""){
    monitor_MQTT();
  }
  else {
    Serial.print(F("MQTT Pub is ")); 
    Serial.println(command); //For Testing
    if (command == "Z2_ON"){
      Z2_ON();
    }
    if (command == "Z2_OFF"){
      Z2_OFF();
    }
    command = "";  //Reset String command and restart loop
  }

}

Sorry PaulS, there is still a sting in there - should that be a character array?

This is a pretty obscure topic so I'm not surprised that no one has encountered it. At the risk of straying from the Arduino topic, if anyone has a similar problem you can achieve the same outcome by calling a python script from HA.
If anyone does find a way to simultaneously monitor MQTT and POST http – please update this topic.

Posting into this very old thread because it was one of only a few returned by Google that covered the exact problem I was having - namely that I couldn't use a HTTP client and MQTT client on the same ESP8266 - but didn't actually have a solution in it.

Simply, the solution is that each wifi client can only maintain a single TCP session, so if you want two of them, you need two wifi (or, in your case, ethernet) clients. i.e.

WiFiClient wifi_1;
WiFiClient wifi_2;
HttpClient httpclient = HttpClient(wifi_1, serverAddress, port);
PubSubClient mqttclient(wifi_2);

No other changes are needed (I dare not post any more code here or someone might take offense at my variable naming, excessive use of String class or tabs!) and this resolved my problem entirely. Hopefully it will help the next person that finds this thread.

Credit to this thread here: TinyGSM disconnects from MQTT broker when it wants to make a GET request · Issue #310 · vshymanskyy/TinyGSM · GitHub

4 Likes

Thank you for sharing the solution! :slight_smile: