I just did it without them and it works just fine. Should I worry about these parts?
You're talking about on the crystal? They're there to ensure the crystal oscillates. There may be enough parasitic capacitance that it happens anyway, but I wouldn't count on it. Otherwise you'll end up with a CPU clock that doesn't tick, nor tock.
If you have the chip plugged into a solderless breadboard, you may have nearly that much capacitance between the lines on the breadboard.
The 22pF capacitors are part of the feedback for the oscillator. It is called a Pierce Gate Crystal Oscillator.
http://www.mpdigest.com/issue/articles/2008/mar/crystek/default.asp
Also each crystal is cut to oscillate with a specific load capacitance - if the capacitance
is a different value the frequency will be somewhat out.
Note that the load capacitance contributed by two 22pF caps is 11pF since they are
effectively in series (from the crystal's point of view).
Many crystals are rated for 18pF load or so, 11pF from the caps, a few pF each from
the wiring and the chip itself.
[ A more in-depth explanation would mention the closely spaced series and parallel resonances of a crystal and the risk of oscillation on a harmonic - one of the symptoms
of missing out the caps could be the oscillator running about 2x or 3x as fast, leading
to total failure of the system to run ]
That article made me feel really ignorant. 
Thanks for the link poly. That one goes in my bookmarks to read and re-read until it becomes obvious.
We're all ignorant about something. We learn the things we need and/or want to know.
Like everything else, the more you learn about electronics, the more you realize you don't know yet.