Delay isn't helping but this piece is a show stopping problem:
while (buttonPresses % 2 == 0) {
for (servoPos = 0; servoPos <= 180; servoPos++) {
servo.write(servoPos);
delay(20);
}
for (servoPos = 180; servoPos >= 0; servoPos--) {
servo.write(servoPos);
delay(20);
}
break;
}
If buttonPresses %2 is 0, that loop will never terminate because there's nothing in there that will ever change buttonPresses. Try replacing the while with an if.
Edit: spelling
Edit two: Didn't notice the break statement. Weird thing to do but invalidates my analysis -- ignore.