I've never used the ESP32.
How do I convert the 1766 into 3.3V?
That doesn't seem right, but divide the ADC reading by 538 (1766/3.3).
The 12-bit ADC should max-out at 4095, and with a 3.3V reference you should read 4095 at 3.3V.
Your voltage divider calculations are right. But, it's usually a good idea to give yourself some "headroom" in case the battery happens to go over 4.8V.