The big question is whether you need that USB interface to the PC. If you do, you might as well shop for a cheap "Freeduino" board like the ones sold by NKC.
If not, all you really need is the CPU chip, a crystal or resonator, and a few caps and resistors. Some people just solder a socket and a few parts to a stripboard or other prototyping board, and transfer the CPU chip from the Arduino board to the project. A search for "stripboard arduino" should get you some links to those.
For permanent projects that don't need a shield, I like the RBBB
: it's a cheap way to get the "core" functions in a compact, easy-to-build form, and the single-row headers make it easy to plug into a larger assembly.