ON and OFF (and sleep?)

The Arduino is based on hardware with very good sleep ability. If you bypassed the regulator, removed the power LED and the FTDI chip, you could get a power efficient solution by reading the atmega datasheet regarding sleep modes. You'd want to run from three AAA cells or similar instead of 9V, so you could run the Arduino from 4.5 to 3.0 volts directly.