And/or I'm gong to guess the input is floating (no connection at all) when the switch is open.
You need a pull-up or pull-down resistor as shown in the [u]Digital Read Serial Example[/u].
The easiest solution is to use the [u]optional-internal pull-up resistors[/u]. But, you'll have to re-wire the switch so pulls the input down to ground, and you'll have to reverse your program logic so that input-low turns the LED on.
[]BTW -[/b] You don't have to turn the LED off after the delay because the loop is going to immediately start-over, read the switch again and turn the LED on or off depending on buttonState.
...And, do you really need the delay? Or, if you want to blink the LED you'll need another delay after you turn it off (and you'll want more than 10mS).