I think the problem is more likely to be latency on the SPI display rather than inaccuracy in the crystal or millis routine.
Try the test with an LCD with parallel connection using the LiquidCrystal library.
or for a simple test, write a sketch that blinks an LED every ten minutes (longer would be better) and compare the blink intervalto the time on your stopwatch.