Why is my servo movement so delayed?

I am using a switch to get my servo go from a certain position to 180 degrees. my code is successful, however when I switch my switch up or down there is about a 20 second delay until the servo actually will move!
here is my code:

#include<Servo.h>
int pos = 0;

Servo servo;
void setup() {
pinMode(4, INPUT);
pinMode(5, INPUT);
servo.attach(9);
}
void loop() {
if (digitalRead(4) == HIGH && pos < 180) {
pos++;
servo.write(pos);
delay(10);
}
if (digitalRead(5) == HIGH && pos > 0) {
pos–;
servo.write(pos);
delay(10);
}
}

You have not posted the code correctly and there is nothing provided about the wiring. The code is not likely to be the problem, it looks satisfactory to me if the wiring matches. No details were provided so no answer is possible. My guess is that pullup resistors are missing.

At a guess, try replacing

  pinMode(4, INPUT);
  pinMode(5, INPUT);

by

  pinMode(4, INPUT_PULLUP);
  pinMode(5, INPUT_PULLUP);

I tried that and it did not work my switch has a middle pin that is wired to 5V the pin above is connected to digital pin 5 the one below is connected to digital pin 4.

Given OP is testing against HIGH To see if button is pressed, it is not an INPUT_PULLUP but indeed maybe OP did not wire the pull down resistor that would be needed... do you have a pull down? (if not use pull-up and change connexion to ground / test against LOW)

The code will go pretty fast as well given the small 10ms delay so if you press the button for 1 second your servo will have done 100 steps... that's pretty fast...

Two pulldown resistors are needed. One resistor goes from pin 4 to ground. The other resistor goes from pin 5 to ground. Each resistor may be any value from about 1K (1000 ohms) to about 10K (10000 ohms).

An Arduino may have a pin mode of INPUT_PULLUP but not INPUT_PULLDOWN. This is why many people who wire switches to their Arduino wire the switch to ground (not to +5) and then expect the switches to be LOW and not HIGH. It saves on wiring up extra resistors.