Change INPUT_PULLUP ?
No. That is keeping the input pin at a known HIGH state when the button is not pressed which is what you want to avoid stray voltages affecting the input.
Changeif (digitalRead(2) == HIGH)toif (digitalRead(2) == LOW)
Note that I am assuming that one of the connections to the button is connected to Arduino pin 2 and the other one is connected to GND. Is that how it is wired ?