Try a little longer delay or search for "Arduino debounce".
You might want to ignore the button release... When you release the button it's probably going to bounce on & off again and another delay (and ignoring the "fake" button presses) after it's released should help.
Or in some cases, the initial button press is ignored, then there's a delay and action is taken then the button is released. (Mouse clicks work that way... Nothing happens 'till you release the button.)