Ok, back again with much better news!
Setting the BOOTRST Boot Reset fuse bit to false prevented the power reset bootloader activation, but it basically disabled the bootloader entirely which just changed the problem.
Luckily my board had a Pulldown resistor on the HWB pin, so after setting the HWBE Hardware Boot Enable fuse bit to true, now the bootloader will activate when I press my reset button(tied to !Reset Pin) and it does not activate under temporary power loss. YAY!
Now my design will be more stable while still allowing USB updating. That was a fun learning experience. Thanks for the input everyone.
Writing fuses ...
LFuse = 0xFF
HFuse = 0xD1
EFuse = 0xFB
Lock byte = 0xEF
Clock calibration = 0x63
Done.