arduino with internal oscillator on 1Mhz

I've got a stand alone battery prowered 328 project that runs off 3 AAs - it "works ok" but reading here is sounds like I'm on the edge of it running successfully at 16MHz.

I was considering dropping the crystal down to 8MHz to give me a wider power margin - Comms is not an issue as I'm programming the 328 in an Arduino then taking it out and dropping it in my PCB.

But it sounds like just changing the crystal won't work either and it will require a software tweak too - is that correct?

Cheers