The ESP32 contains an internal RTC and has APIs supporting standard POSIX time functions. So, I'd do all time readings in the code from the internal RTC and just read the external RTC once at startup and then every couple of hours to true up the internal one.