A linear deviation from the wall clock seems to make sense if the oscillator is off frequency OR if you have a bug that introduces a consistent error.
I would think that no matter how many times you sample the ADC, that ints would be enabled between each call. A ~2mS SPI ISR time sounds unlikely.
At this point, I would do what it takes to find out precisely (as you can) what your resonator is running at on one of your boards by running a time keeping routine, like I described, for 24 hours and just see how far off it is. For every second that it's in error after 24 hours, that's roughly 10ppm of error. Or if you have an accurate scope or frequency counter? Once you know what your base error is, then we can determine if your results are to be expected or if they are out of line because of a bug.
EDIT: If you have an RTC or even a GPS that can generate a 1pps pulse, I can give you a sketch that will precisely measure the pulse length (1uS) using the hardware capture. You can then take this measurement and get a decent idea of how far off your resonator is. For example if you get consistent measurements of a one second GPS pulse of 999700uS, then you know your off by 300ppm.