@jrsikken, either change the boards.txt file then burn the bootloader (which also burns the fuse bytes) or just change the one fuse byte with AVRDUDE.
BTW, the top five bits of the efuse sort of "aren't there", so setting the efuse to 0x04 turns out to be the same as 0xFC.