The logic should be fairly simple - like this pseudo code
void loop() {
IRstate = read IR sensor;
btnState = digitalRead(btnPin);
if (btnState == LOW && IRstate == OFF) {
myServo.writeMicroseconds(1250); // choose a number for a suitable speed
}
if (IRstate == ON) {
myServo.writeMicroseconds(1500); // or whatever value causes the servo to stop
}
}
Using servo.writeMicroseconds() gives you finer control.
I assume the button pin uses INPUT_PULLUP so that LOW = button pushed
I don't know whether the IR device gives HIGH or LOW for on so I have just used OFF and ON