Real time clock a waste of time ?

The purpose of the RTC chip is not to be more accurate than the crystal clocked Arduino.

When you turn off or reset your Arduino, the millis counter starts from 0 when you power up again. The RTC takes a battery back up and continues to keep time even with the Arduino off. Your way, how do you go from your millis counter to real time and date without having to set the clock every time you power up?