If it's a design you're giving out, or especially selling pre-manned boards, I would vote for having that ground pin hole. It may work without, but you're probably out of spec on the ideal load capacitance of your oscillator. And after all, it's harmless if you don't use it yourself.