bsld:
Also what will happen if I press the button before the 'setup' function executes or if I mistakenly don't configure the pin as INPUT?
Arduino pins are defaulted to inputs, which is a safe state.
Consider wiring two inputs to each other. Nothing happens as both expect but don't receive.
Consider wiring two outputs to each other. The pins are now trying to source current into each other which is bad.
Default inputs is safe.
Also if your pushing the button prior to the setup function, the pin will sink current, but nothing will happen. The MCU is safe and will wait for your code to tell it to do something with it's current pin state.
https://www.arduino.cc/en/Tutorial/DigitalPins
This explains further on pins and why pull-ups/pull-downs are necessary.