This question has been asked many times and this a common rookie mistake.
So basically when the pin STATE is zero, the LED should be off.
and when the pin STATE is high or 5V, LED is ON.
The problem you're facing is, when the input pin drops low, the LED still stays ON, sometimes flickers, DIMS etc. The reason is, the INPUT pin on LOW state is not completely LOW. On LOW state the voltage should be 0V, but usually the pin picks up noise from the environment and the PIN state fluctuates even though the PUSH button is not pressed.
Solution #1 :
pinMode(inputPin,INPUT); //This you already have in your code I believe
digitalWrite(inputPin,HIGH); //This helps to enable pull down resistors in the chip to make sure the input pin's state drops down to a perfect zero or LOW once you release your push button. This usually works fine.
//The rest of your code is the same.
Here, there is no change in code at all. but a 10K or a 20k resistor. I believe your connections are like this ryt now
5v pin ---->PUSH BUTTON ---->INPUT PIN
One addition here,
5v pin ----PUSH BUTTON ---> INPUT PIN
Ground PIN ---> 10k or 20 k resistor -->INPUT PIN
The input pin is common here. This makes sure the pin drops to LOW fully when the button is released. Don't worry, even though the ground and 5 v pins are connected to the same input PIN, the resistor makes sure you don't short them.
Try soln #1, then if it still malfunctions, revert your code and buy that 10k resistor $)