One "arduino millisecond" IS NOT equal to 1024 real milliseconds!!! Whether you reset your timer for each "alarm" does not change anything.
The problem is that the oscillator is not completely accurate nor is it temperature compensated. You cannot use the on-board (or on-chip) oscillators for accurate timing and that's just the way it is.
You should get a temperature compensated RTC module which allows you to calibrate the aging register (DS3231). If calibrated well, such a module will only loose or gain less than a second per month!