I had the same problem, already now for a long time (years?)
My devices were (usually) not showing up in the 'Tools → ‘Port’ menu of the Arduino IDE. However, very occasionally (let’s say 1 in 50 times), they would show up (or at least some of them). Very strange!
I am using Windows 7 SP1, version 2.6.3 of the esp8266 board package, and Arduino IDE 1.8.11 . But the problem was already there in earlier versions of the IDE and earlier version of the esp8266 board package. I seem to remember that long time ago, with a much earlier version of the IDE, things were working fine.
I checked everything over and over. All seemed fine. I followed most of the hints and tips as given in this forum, and sometimes things would work, but sooner or later it would stop working again.
To look around on my own network, I downloaded and installed ‘Bonjour browser for Windows’. This program shows my ESP devices, nicely advertising their OTA update port, 8266 . Now, I noticed that, while running this ‘Bonjour browser for Windows’ program, the Arduino IDE would suddenly show my devices in the 'Tools → ‘Port’ menu. It was working! So I thought. The next day, the devices would not be shown in the Arduino IDE.
I suspected the running of ‘Bonjour browser for Windows’ simultaneously with the Arduino IDE had something to do with it. So I fired that up again, and lo and behold, after a few moments the devices were shown in the Arduino IDE!
Repeated a couple of times, and it was consistent: when ‘Bonjour browser for Windows’ was running, the devices are shown, otherwise not.
So I decided to have a look with ‘Wireshark’ (a network packet sniffer). I captured all ‘mdns’ packets during a session when ‘Bonjour browser for Windows’ was running together with the IDE. I then compared that to a similar session with ‘Bonjour browser for Windows’ not running. In the latter session (only the IDE running), I noticed that my laptop was not sending any “Standard query” packets to port 5353. When running the Arduino IDE together with the ‘Bonjour browser for Windows’, I could see the “Standard query” packets to port 5353 being sent and all devices responding one by one, and subsequently appearing in the IDE’s menu.
Being completely convinced that this “OTA” stuff was specific to the esp8266 hardware (Arduino Uno does not support it), I searched through all the code and (Python) scripts in the directory structure of the ‘esp8266\hardware’ board package directory. But I found nothing that I could relate to mDNS.
So I decided to look further, in the files of the Aruino IDE itself, in/under ‘c:\Program Files (x86)\Arduino-1.8.11’. In the ‘…\lib’ subdirectory, there is a file called ‘jmdns-3.5.3.jar’. That looks like something to do with mDNS. After Googling for it, I found there is a newer version of this Java library, at https://mvnrepository.com/artifact/org.jmdns/jmdns/3.5.5. So I downloaded the ‘jmdns-3.5.5.jar’ file, removed ‘jmdns-3.5.3.jar’ out of the ‘…\lib’ subdirectory and placed ‘jmdns-3.5.5.jar’ in it.
As a result, running the ‘c:\Program Files (x86)\Arduino-1.8.11\arduino.exe’ would crash Somehow, the ‘arduino.exe’ expects exactly version 3.5.3 of this .jar file.
I loaded ‘arduino.exe’ into a hex (binary) editor, looked around a bit and indeed I found he hard coded file name with the version in it. Keeping a copy of ‘arduino.exe’ in ‘arduino.exe.bak’, I simply changed the string ‘jmdns-3.5.3.jar’ to ‘jmdns-3.5.5.jar’ (see attached screen capture).
Suddenly it all worked. I could launch the Arduino IDE and it would consequenly show all my devices, directly at startup. No need for other software or ‘Bonjour service’ to run along. Another trace with ‘Wireshark’ now shows nicely the “Standard query” packet to port 5353, and all the responses, when starting up the Arduino IDE.
So my conclusion: the Aruino IDE is using a buggy version 3.5.3 of the Java ‘jmdns’ library, and upgrading that to version 3.5.5 solves (for me) the issues.
To double-check, I switched back to version 3.5.3 of the Java ‘jmdns’ library, and indeed the flaky behaviour of the devices not showing up in the IDE started all over again.
One strange thing I noticed: on my desktop PC, all was working fine, even with the version 3.5.3 of the Java ‘jmdns’ library. So I decided to dig deeper. My desktop PC has only one network interface, my laptop has multiple (wired, wireless, even a couple of “virtual” and “VPN” ones). So I setup Wireshark to start capturing on all network interfaces simultaneously (yes, Wireshark supports that ). Now I could see clearly what was happening. With version 3.5.3, the “Standard query” packets to port 5353 are sent to only one network interface (the wrong one). With version 3.5.5, the “Standard query” packets are sent to all (“connected”) network interfaces. The responses are then coming in on the active network interface, and the devices are subsequently shown in the Arduino IDE menu.
I had a look in the other versions of the Arduino IDE. The current newest version, 1.8.12, is still using 3.5.3. Version 1.8.5 and older were using 3.5.1 . This corresponds to forum postings reporting that IDE versions 1.8.5 and earlier are working ok. I also checked the sending of the “Standard query” packets; indeed up to version 1.8.5 they are sent to all network interfaces, and starting from version 1.8.6 they are sent to one (usually wrong) network interface.