You need a regulated power supply, which could mean putting a voltage regulator somewhere in your circuit (unless you have another source for it).
In the current Arduino core a few things don't work if you're not using an external oscillator at 16 MHz. I'm planning to fix this for Arduino 0010 and the Arduino LilyPad (which will be using the internal clock of the ATmega168).
A reset button is often convenient, though you can just tie the reset pin high and remove power to the board to reset it.
An ICSP header (the six male pins on the Arduino board) can be useful, but you don't need them if you don't mind moving the ATmega to another circuit to reprogram it.