Hey Brian! Not quite. Now I can't speak for the ATMEGA directly since my experience is with PICs but if I want to take my microcontroller off of the development board, I have to set certain pins high (MCLR in my case). Also, when you use the arduino IDE, your duemilanovae code will most likely be programmed into the chip assuming that there's an external oscillator attached (16 MHz). So if you just apply voltage, you won't get results you will want.
Sparkfun has a nice little tutorial on making a "breadboard" Duemilanovae/Uno using the Uno without the ATMEGA328. Take a look, and you can strip off the parts you won't need. Basically, you probably won't need the ICSP once you programmed the chip, or the buttons, power regulator etc... You decide what you want to keep.
http://www.sparkfun.com/datasheets/DevTools/Arduino/SFE03-0010-RetailMedium-BBAC-guide-WEB-01..pdf