If you use a DC converter to reduce > 5V to 5V, then you will lose less current than if you use the onboard linear regulator.
Will you be wanting to recharge the battery as part of the project?
Long ago we had things that when you pushed the power plug in, it moved a physical switch that turned battery power off. We didn't have the low-current devices we have now, but the built-in switch worked.