There's no worries about the ATmega328 itself being programmed on 5V even if you're expecting to run it at 3.3V. But like you said, the concern is with whatever other components would be exposed to the 5V from the programmer. I use Adafruit's USBtinyISP which has a jumper to optionally prevent the programmer from putting power to the ICSP header (in which case the MCU will need to be powered via its intended 3.3V supply).Something similar could be done on the target board itself, i.e. a jumper which when pulled disconnects everything but the MCU from the supply.
Does anyone have any recommendations or procedures for accomplishing something like this?
Well I had started to write that all should be fine, then a light went on. The datasheet says that the voltage on any pin (except RESET -- for high-voltage programming) cannot be less than -0.5V or more than Vcc + 0.5V. So I wouldn't drive an MCU running on 3.3V with 5V signals. I'd either level-shift the programming signals or why not just run the programmer on 3.3V too. I've done ICSP programming at 3V and 3.3V as well as at 5V, flash as well as EEPROM should work down to 1.8V per the datasheet. I'd breadboard it up just to convince myself, but that's the way I'm reading it. I'd seriously lean toward running the programmer at 3.3V if possible, but if not, bi-directional level shifting seems like overkill. SCK and MOSI are inputs to the target, MISO is an output, etc., so one-way shifting should suffice.HTH!