A button is a resistor with either infinity resistance or with zero resistance.
From the datasheet of the 328P
If you enable the internal pullup, you open the FET and the signal Logic will be Vcc if no button is connected or if the button is not pressed. A button wired between Pxn and GND and pressed will short the Logic signal to ground.