ESP8266 + Arduino Uno + MQTT Problems. So much conflicting information

I am trying to use a ESP8266 + Arduino Uno + have it communicate to MQTT. My goal is for someone to click a button attached to the arduino that then turns on an LED attached to a second arduino many km away. To accomplish this my current hypothesis is to have the arduino communicate with an ESP8266 to an MQTT broker, however, I can’t for the life of me figure it out. There are so many conflicting reports on how to wire them together, how to code them, what settings to use, what is necessary/what isn’t. I keep getting dozens of different errors depending on what I do. So here is where I am. I’m pretty sure This is a thread that addresses my problem or at least is similar that I could branch off of.

Here is my code

#include <ESP8266WiFi.h>
#include <PubSubClient.h>
 
const char* ssid = "***";
const char* password =  "***";
const char* mqttServer = "***";
const int mqttPort = ***;
const char* mqttUser = "***";
const char* mqttPassword = "***";
 
WiFiClient espClient;
PubSubClient client(espClient);
 
void setup() {
 
  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.println("Connecting to WiFi..");
  }
  Serial.println("Connected to the WiFi network");
 
  client.setServer(mqttServer, mqttPort);
  client.setCallback(callback);
 
  while (!client.connected()) {
    Serial.println("Connecting to MQTT...");
 
    if (client.connect("ESP8266Client", mqttUser, mqttPassword )) {
 
      Serial.println("connected");  
 
    } else {
 
      Serial.print("failed with state ");
      Serial.print(client.state());
      delay(2000);
 
    }
  }
 
  client.publish("esp/test", "Hello from ESP8266");
  client.subscribe("esp/test");
 
}
 
void callback(char* topic, byte* payload, unsigned int length) {
 
  Serial.print("Message arrived in topic: ");
  Serial.println(topic);
 
  Serial.print("Message:");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
 
  Serial.println();
  Serial.println("-----------------------");
 
}
 
void loop() {
  client.loop();
}

and here is my current error

Arduino: 1.8.9 (Mac OS X), Board: "Generic ESP8266 Module, 80 MHz, Flash, Disabled, All SSL ciphers (most compatible), ck, 26 MHz, 40MHz, DOUT (compatible), 512K (no SPIFFS), 2, nonos-sdk 2.2.1 (legacy), v2 Lower Memory, Disabled, None, Only Sketch, 115200"

Sketch uses 275428 bytes (55%) of program storage space. Maximum is 499696 bytes.
Global variables use 27284 bytes (33%) of dynamic memory, leaving 54636 bytes for local variables. Maximum is 81920 bytes.
esptool.py v2.6
2.6
esptool.py v2.6
Serial port /dev/cu.usbmodem14101
Connecting........_____....._____....._____....._____....._____....._____.....____Traceback (most recent call last):
  File "/Users/***/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.2/tools/upload.py", line 25, in <module>
    esptool.main(fakeargs)
  File "/Users/***/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.2/tools/esptool/esptool.py", line 2653, in main
    esp.connect(args.before)
  File "/Users/***/Library/Arduino15/packages/esp8266/hardware/esp8266/2.5.2/tools/esptool/esptool.py", line 468, in connect
    raise FatalError('Failed to connect to %s: %s' % (self.CHIP_NAME, last_error))
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header
esptool.FatalError: Failed to connect to ESP8266: Timed out waiting for packet header

If I’m going about this wrong please let me know, I have attached a photo of my setup

The tutorial, your code and your Tools > Board selection all assume you’re programming the ESP8266 directly. However, if your goal is to use an ESP8266 to provide WiFi connectivity to an Arduino Uno then this is very likely the wrong approach. For that usage, most people will run the stock AT firmware on the ESP8266 module. Your ESP-01 module comes pre-programmed with this firmware. They will write a sketch for the Uno that controls the ESP8266 over a serial connection.

Another option is to leave out the Uno altogether and just use the ESP8266 as an Arduino board. In some ways (clock speed, memory), the ESP8266 is a superior microcontroller to the ATmega328P on the Uno. The ATmega328P has it’s own advantages (5 V compatible, more I/0, more ADC, easier to use, better support), but the ESP8266 is certainly This is the usage the tutorial and your code is written for. However, your ESP-01 is not a great choice for this application. It can be done, but you will need to provide a power supply and a USB to TTL serial adapter. You can buy reasonably priced ESP8266 boards that have all that in a single package and also break out all the GPIO pins of the ESP8266 (your ESP-01 only breaks out a few of the pins. A popular choice is the WeMos D1 Mini, but there are several good options. In the case of people who do want to program the ESP-01 module directly, some will use an Uno as a USB to serial adapter. It’s more convenient to use a dedicated USB to serial adapter, but it is possible to use the Uno for this purpose.

It is possible to program the ESP8266 with a custom firmware and still connect it to an Uno that uses whichever communication method you have written into your ESP8266 firmware. However, this adds more complexity to the project so it’s probably better to just decide on one of the two options I’ve explained above to get started.

Given this information, please provide a more detailed description of what you want to accomplish.

pert:
The tutorial, your code and your Tools > Board selection all assume you're programming the ESP8266 directly. However, if your goal is to use an ESP8266 to provide WiFi connectivity to an Arduino Uno then this is very likely the wrong approach. For that usage, most people will run the stock AT firmware on the ESP8266 module. Your ESP-01 module comes pre-programmed with this firmware. They will write a sketch for the Uno that controls the ESP8266 over a serial connection.

Another option is to leave out the Uno altogether and just use the ESP8266 as an Arduino board. In some ways (clock speed, memory), the ESP8266 is a superior microcontroller to the ATmega328P on the Uno. The ATmega328P has it's own advantages (5 V compatible, more I/0, more ADC, easier to use, better support), but the ESP8266 is certainly This is the usage the tutorial and your code is written for. However, your ESP-01 is not a great choice for this application. It can be done, but you will need to provide a power supply and a USB to TTL serial adapter. You can buy reasonably priced ESP8266 boards that have all that in a single package and also break out all the GPIO pins of the ESP8266 (your ESP-01 only breaks out a few of the pins. A popular choice is the WeMos D1 Mini, but there are several good options. In the case of people who do want to program the ESP-01 module directly, some will use an Uno as a USB to serial adapter. It's more convenient to use a dedicated USB to serial adapter, but it is possible to use the Uno for this purpose.

It is possible to program the ESP8266 with a custom firmware and still connect it to an Uno that uses whichever communication method you have written into your ESP8266 firmware. However, this adds more complexity to the project so it's probably better to just decide on one of the two options I've explained above to get started.

Given this information, please provide a more detailed description of what you want to accomplish.

A lot of confirming things you wrote! With so many different tutorials in how to use the esp8266 with arduino I thought perhaps I was looking at the wrong usage case. I've got a NodeMCU being shipped to my place right now. If I used that would it fix a lot of my problems by starting from scratch? This is my general goal.

When button A attached to microcontroller A is pressed. It sends a signal to microcontroller B many km away to turn on LED B that is attached to microcontroller B. It goes both ways too. So when Button B that is attached to microcontroller B is pressed, it sends a signal to microcontroller A many km away to turn on LED A.

My current hypothesis is to have this communication over an mqtt network.

guanciale:
If I used that would it fix a lot of my problems by starting from scratch?

The NodeMCU is similar to the WeMos D1 Mini, it's just a little bigger. It has the USB to TTL serial adapter and power supply on board so it's very easy to use as an Arduino board. As long as the driver for the USB to TTL serial chip is installed, can just plug it in to your computer and upload sketches to it using the Arduino IDE. You can easily connect a button directly to one NodeMCU and an LED to the other one, so there is really no need to involve an Uno in your project. However, if you want to do it with the Uno that's certainly possible.

guanciale:
My current hypothesis is to have this communication over an mqtt network.

I assume both ends of the system will be connected to a WiFi network that can connect to each other using a LAN cable or Internet access?

MQTT would be okay but you need yet another system/PC to host the MQTT broker.

For such a simple system you might be better just using direct connection between ESP's (if on LAN) or using an internet service like dweet.io to pass messages back and forth.