The code you've posted ought to work OK the very first time the button goes HIGH after a reset, but after that you're going to hit various bugs.
The most likely reason for the big delay you mention is that you have already pressed the button previously and the sketch is still trying to sweep the servo from the previous operation. The bugs in it will stop that doing anything useful most of the time, but it will still keep the sketch busy and stop it detecting further inputs for a long time.
Do you specifically want to move the servo slowly?
Do you really need to detach the servo when it has reached the end position, and if so why?
Your code to return the servo to zero degrees is missing.