@Pluggy: I actually bought a DSO, a frequency counter and an efratom time normal (in order to ensure proper overkill
). Conclusion: my code was off by one cycle (=1/16 ppm). The rest was due to the crystal setup. I discussed this with a professional electrical engineer. He explained to me that with the arduino like setup it may be possible to get into the 100ppm range. But this is not at all reliable. The main issues are temperature, voltage and of course aging.
True: the original Arduino did not drift that much at room temperature. However if you need guaranteed timing better than 100ppm, a 100ppm crystal is not at all sufficient. You will need a VXCO, TXCO or something along these lines. It also seems that dedicated time keeping chips (based on 32kHz crystals) offer much better precision than the standard setup and they are comparatively cheap (compared to TXCOs).
There is one more thing I learned by now: crystal oscillators are mechanical devices and thus the frequency may drift in the presence of vibrations as well.