I did not think of VPN, that would solve the problem of the LANs but would (probably) make the delay even higher...
I already tried this with the library ESPAsynchWebServer and it is based on HTTP requests, so it is even slower than basic TCP.
This would be surely easier, but my application works on the idea that one ESP32 receives the information from the other almost instantaneously.