That is probably because of bouncing. Have you tried to use a button library?
https://www.arduino.cc/reference/en/libraries/ebutton/