find myself unplugging the usb cable every time need to make a hardware change.
That's what you are supposed to do. The best way of blowing up your board is to work on it while it is powered up. If you want a switch then power it from an external PSU and switch that.