Raw UDP is your best bet. Maybe send 3 duplicate packets instead of one to account for any packet loss that may happen in transit.
I have a setup where my MCU talks to an ESP8266 module over UART at 921.600 Kbps (AT commands) to transmit UDP and my "send UDP" state machine takes about 6ms to complete with a packet payload of about 32 bytes. You should be able to achieve better than that with a newer ESP32 and direct programming.