There are several problems I see here. I see no pullup on the input pin, and I see no debouncing. In addition, as already pointed out, the if-statements make no sense.
I would have done it as
int last_state = 2;
int current_state = digitalRead(BUTTON);
if(current_state != last_state)
if(current_state == digitalRead(BUTTON))
last_state = current_state;
digitalWrite(LED, last_state == 0 ? LOW : HIGH);
The problem with all switches is that they are mechanical. This means the contacts "bounce". Here's what you think happens:
In this diagram, the input value of the pin is low (...); the button is pressed and the input goes high (|||||) and the switch is released and it goes low again. But that's not what really happens. What happens is this:
As the switch makes contact, the physical mechanism does not make a solid connection, and you get some ups and downs before it settles in to being high. And when you release it, it may drop to GND and back up to +V a few times before it settles into being GND. It usually settles down in 10ms or so that's why the 10ms delay. You say "But I don't see the light flicker". Yes, our perceptual system is far too sluggish to detect this. But the computer isn't. In a few milliseconds, even the poor 16MHz Arduino AVR chip can read those pins hundreds, maybe even a couple thousand, times. It sees every bounce. Now, if it was supposed to COUNT the number of times is saw the button pressed, it might tell you a much larger number without the debouncing. In one demo I did, a single button press added 23 to the counter, by far the largest amount I ever saw. But a typical button press added 3 to 7 counts.
I do not see a pullup or pulldown in your circuit. A pullup resistor is a large-value resistor which connects the pin to +V. In this configuration, the button connects the input pin to GND. There is a similar mechanism called "pulldown", where a large-value resistor connects the pin to GND, and the button connects the pin to +V. In input, the pin must NEVER be in an "unknown" state, that is, not connected to a known voltage level. If it is not connected, you have no idea what you are going to read. Electrical noise will generate spurious inputs.
For an AVR chip, you should set the mode to INPUT_PULLUP, which connects a 30K-50K ohm resistor from the input pin to +V entirely within the chip, so you don't need an external pullup. Note that in this configuration, when the button is pressed, "pressed" reads as 0 and "released" reads as 1. In the case of the pulldown, pressed reads as 1 and released reads as 0, but for AVR chips, this requires an external resistor, so it is rarely used. Other chips have pulldown resistors as well. But I don't see how the circuit you have described could work. If you set BUTTON to INPUT_PULLUP and not INPUT, and invert the sense of the value, such as by changing the conditional to (last_state == 0 ? HIGH : LOW), it should work.
In my class, I present an SPDT switch, with one point wired to +V, the other to GND, and the center to the input pin. I ask "Will this work?" They usually say "yes" because the pin is always connected to a known voltage. Not true. The correct answer is "no", because there is a period of a huge number of microseconds when neither end is connected. If the pin is ever not connected, solidly, to a known voltage source, you will see behavior that looks like
Where the * symbols indicate random voltages picked up by the unconnected pin. Now, if you wrote code that turned the LED on with one push and turned it off with another, when you pushed the button with no debouncing, you would have no idea what state it would end up in. And if you have no connection to the pin, and no debouncing, the LED would go on and off randomly as each noise transition was reported as a button press