Its hard enough to just get the thing to count down 5 minutes from when the arduino turned on never mind accounting for the time millis() started in that whole mess.
It sounds to me like you're over-thinking this.
A 32 bit subtraction in the microcontroller can be performed many tens of thousands of times a second.