I suggest that you look into using a finite state machine. States could be "move CW", "move CCW" and "wait 5 seconds". Each state would have its own millis() timer (like the several things at a time example).
Run the servo state machine in parallel with reading and acting on the switch.
For more info just google "arduino state machine",