I wasn't aware of the Prescaler library. I have just adjusted it myself, it takes about four lines of code. My approach has been to set up entries in boards.txt for clock speeds of 1, 2, 4 and 8MHz so that sketches get compiled with the intended F_CPU. If then my hardware setup doesn't match the intended F_CPU, for example using a 16MHz crystal but I want a 2MHz system clock speed (and compiled with the 2MHz boards.txt entry), then the first thing I do in setup() is to make the appropriate prescaler adjustment.
This approach allows millis() and delay() to work properly. Also I have had no issues with my Button library.
If you don't mind sharing, tell me a little more about what you're trying to do, and which microcontroller you are using. Supply current while sleeping should be measured in µA, not mA. Using power-down sleep mode with the BOD disabled, current should be a fraction of a microamp.