Without the right load caps the frequency won't be in-spec for that crystal (they are calibrated for a given load capacitance).
Load caps make the oscillation frequency much less sensitive to stray circuit capacitances I believe - not necessarily important.
A particular oscillator design may assume a certain range of load capacitance too - without it there might be no guarantee of oscillator start-up?