Controlling a servo with a single ON/OFF toggle button

I have been doing a project and need some help controlling a servo with a single button. I want it to sweep, then when I press the button it stops at the position it is in. When I press the button again, it will continue sweeping at any position it is in. Down below is the code I have.

#include <Servo.h>
int button = 5; //button pin, connect to ground to move servo
int press = 0;
Servo servo;
int pos = 0;    // variable to store the servo position
boolean toggle = true;

void setup()
{
  pinMode(button, INPUT); //arduino monitor pin state
  servo.attach(6); //pin for servo control signal
  digitalWrite(5, HIGH); //enable pullups to make pin high
}

void loop()
{
  press = digitalRead(button);
  if (press == LOW)
  {
    if(toggle)
    {
  for (pos = 0; pos <= 90; pos += 1) // goes from 0 degrees to 90 degrees
                                      // in steps of 1 degree
    servo.write(pos);              // tell servo to go to position in variable 'pos'
    delay(15);                       // waits 15ms for the servo to reach the position
      
  for (pos = 90; pos >= 0; pos -= 1) // goes from 90 degrees to 0 degrees
    servo.write(pos);              // tell servo to go to position in variable 'pos'
    delay(15);                       // waits 15ms for the servo to reach the position 
    
      toggle = !toggle;
    }
    else
    {
      servo.write(pos);   // servo stops at the position it is in
      toggle = !toggle;
    }
  }
  delay(500);  //delay for debounce
}

If you want the switch to be responsive you have to avoid using delay()s.

Using millis() for timing. A beginners guide
https://forum.arduino.cc/index.php?topic=503368.0

Study how to use a 'State Machine'
http://www.thebox.myzen.co.uk/Tutorial/State_Machine.html

Study 'State Change Detection'