So weird, as soon as I move the RTC reset alarm flag to the main loop, it freezes when it does that. But it works fine in the interrupt loop.
Granted I haven't tried it with the stand alone Arduino. So weird though.
So at least the code in ISR is less, just clears RTC flag and set MCU flag.