Does it behave the same if you power it up by connecting via USB?
(In theory, the bootloader only runs when the reset status register identifies the reset cause as "external reset." "Power-on reset" is supposed to be a separate cause, which should not cause the bootloader to run.)