Barebones ATmega328 - Why use caps on the crystal?

Without the right load caps the frequency won't be in-spec for that crystal (they are calibrated for a given load capacitance).

Depends on what you meant by "in-spec".

Load caps make the oscillation frequency much less sensitive to stray circuit capacitances I believe - not necessarily important.

It doesn't make it less sensitive. It makes the frequency closer to spec.

A particular oscillator design may assume a certain range of load capacitance too - without it there might be no guarantee of oscillator start-up?

Yes, but generally only for low frequency oscillators.