What does the math say? 3 mins in 4 hours is 1.25% slow. That is a couple of magnitudes higher then the normal crystal tolerance error of 100-150 parts per million, so I would not look first at the crystal to be the problem.
My guess is you are dealing with a math problem in your code that is dropping mill counts. But I can only leave that to the software gurus around here, hardware is my thing. 
Lefty