Button Press -> Servo rotates back and forth one time

I'm trying to make a servo sit at 90 degrees, and upon pressing a button have it go to 180, then to zero, then back to sit at 90 until another button press.

I have a working button press -> LED blink setup.
I can also plug my servo in and control it.
The weird thing is that when I try to combine the two, it breaks my LED setup and I have no idea why. Servo doesn't work either in the combined setup.

Here's what happens:

Initial LED Setup- I hit the button and my LED flashes on and off. I plug my servo cables into the PWM slot, ground on the power side, then power to the 5V line on my breadboard, and it doesn't work and my LED setup stops working too. I just rebuilt it and the LED setup worked again (same slots), then when I tried to re-add the servo, it broke the LED setup again.

Any ideas what I could be doing wrong?

Link to code: Button Press + Servo Control - Pastebin.com
Link to wiring: http://i.imgur.com/43TjOCZ.png

You'll need to post the code and your setup for us to help.

Also, a detailed description of what happens. When you say "it doesn't work" that doesn't give us any information - How does it not work? What does happen instead of the expected effect?

I plug my servo cables into the PWM slot, ground on the power side, then power to the 5V line on my breadboard, and it doesn't work and my LED setup stops working too.

What is a "pwm slot"? Servos need an external power supply, don't power them from the arduino. Simple servo momentary button code.

//zoomkat servo button test 7-30-2011

#include <Servo.h>
int button1 = 4; //button pin, connect to ground to move servo
int press1 = 0;
Servo servo1;

void setup()
{
  pinMode(button1, INPUT);
  servo1.attach(7);
  digitalWrite(4, HIGH); //enable pullups to make pin high
}

void loop()
{
  press1 = digitalRead(button1);
  if (press1 == LOW)
  {
    servo1.write(160);
    delay(2000);
    servo1.write(20);
  }
}