timing accuracy test

I'm checking the accuracy of my arduino clock against a (ChronoDot) DS3231 TCXO running a 1Hz interrupt using the attached code. Results show +/-1 ms difference every few seconds. Latency in the code, arduino millis calculation issue or crystal? Please comment.

TIA.

//ChronoDot test [ds3231]
//NOTE: 10k pullup resistor on SQW line
#include <Wire.h>

#define INT_PIN 3  //chronodot sqw wire

extern volatile unsigned long timer0_millis;
volatile unsigned long t, now, prev;

void setup() {
  //init wire & serial
  Wire.begin();
  Serial.begin(9600);
  
  Wire.beginTransmission(104);
  Wire.send(0x0e);  //select control register
  //set sqw at specified Hz of  0=1Hz
  Wire.send(0);
  Wire.endTransmission();
  
  //init interrupt
  pinMode(INT_PIN, INPUT);
  attachInterrupt(1, OneHzInterrupt, FALLING);  //0 = digital pin 3
}

void OneHzInterrupt(void) {
  now = timer0_millis;
  t = now - prev;
  prev = now;
}

void loop() {
  Serial.println(t);
  delay(1000);
}

The way the code is implemented timer0_millis increments by 1 every 1.024ms then there is a "correction" every 41 milliseconds and it actually increments by 2. Another variable timer0_fract increments by 3 until it reaches 125 indicating it counted 24 times out of 1000 (3/125-->24/1000).

Think of it like this table:

| Actual Time | | timer0_millis | | timer0_fract |
| - | - | - |
| 1.024ms | 1 | 3 |
| 2.048ms | 2 | 6 |
| 3.072ms | 3 | 9 |
| ... | ... | ... |
| 41.984ms | 41 | 123 |
| 43.008ms | 43 | 1 |
| 44.032ms | 44 | 4 |
| 45.056ms | 45 | 7 |

Thus the correction leads to a hiccup between 41 and 43.