The oscillator circuit is specifically designed for reliable operation with crystals, and with only a few allowed variations. See the processor data sheet for some of the details.
There are also technical documents on the manufacturer's web site that explain the limitations of the oscillator circuitry more fully.