Page 43 of the Atmega328P datasheet says:
9.10.6 Port Pins
When entering a sleep mode, all port pins should be configured to use minimum power. The most important is then to ensure that no pins drive resistive loads. In sleep modes where both the I/O clock (clkI/O) and the ADC clock (clkADC) are stopped, the input buffers of the device will be disabled. This ensures that no power is consumed by the input logic when not needed. In some cases, the input logic is needed for detecting wake-up conditions, and it will then be enabled. Refer to the section ”Digital Input Enable and Sleep Modes” on page 79 for details on which pins are enabled. If the input buffer is enabled and the input signal is left floating or have an analog signal level close to VCC/2, the input buffer will use excessive power.
This suggests that the input pins are disabled anyway (especially since I do not have any configured to wake it up).
So it would seem you don't need to obsessively put pull-down (or pull-up) resistors on all unused pins, since in the deep sleep modes the pins will be disabled. And if you had one configured as an interrupt to wake it up, presumably that would not be floating.
However I take it as a good point that in some sleep modes, floating voltages on input pins could well be a bad idea.