Gamers minimize latency by using Ethernet instead of WiFi. Use an Ethernet connection directly to the router on both ends. WiFi adds 5-10 ms while Ethernet adds < 1 ms.
If using plain TCP, be sure to disable Nagle's algorithm on both sides.If you want to stick with ESP32 WiFi, I am pretty sure ESP32 has an API function to disable Nagle. If using a library built on top of TCP (for example, websockets and HTTP), search the source code for "nodelay" or "nagle" to see if it a optimized for low latency.