Welcome to the forum
Which board are you using ?
I have been using OTA for a few days for the first time and have had no problems with using the IP address. I currently have 3 devices using OTA, 1 ESP32 and 2 ESP8266. All of them work fine with the IP address allocated to them automatically
What makes you think that OTA is trying to "speak" to the wrong address ?