You can eliminate the crystal and associated load capacitors if you change the fuses to select the internal 8 MHz RC oscillator. You can do this by selecting one of the 8 MHz boards (like "LilyPad Arduino w/ ATmega328") and burning a fresh bootloader.
Just buy the resonator and skip the internal oscillator option
I just don't understand why you don't want to use it in your project....( I don't know your situation... )
FYI in the future... This page will help you setup a chip for use with out the need for external mumbo jumbo.http://arduino.cc/en/Tutorial/ArduinoToBreadboard
Hi If I have grasped it correctly, the capacitors on the power-pins are required to removethe DC ripple effect.