may be because when the servo is not rotating your code is stuck in this delay for 5s
so you are not going back to the loop where you test the button.
➜ get rid of all delays (that are more than a few ms) if you want no perceived lag in user interaction. The loop needs to run fast and never be stopped or blocked somewhere
You need to modify the update() method so that it also knows when to pause for a while in the same way you used millis() and lastUpdate. This could be done by smartly modifying updateInterval for example (but keeping a copy of the original value)