dc42:
The capacitors perform another important function. The combination of capacitors, crystal and the inverting amplifier make up a Colpitts oscillator. Without the capacitors (or sufficient stray and pin capacitance instead), the circuit would not oscillate.
CMOS inverter is a phase-shift oscillator surely?