How warm? Does the regulator on the ESP32 module get warm? (that's the black rectangular chip, fairly large, with 3 pins on one side, and one much wider pin (called the tab) on the opposite side).
Is anything else connected to it.
Does it work with a different 5v power source? Try a different phone charger. I thought the ESP32's sucked enough power to prevent this, but some USB 5v charger devices will sense low current and decide "Okay, device is charged, time to save power by shutting down"