Why didn't you install the 2 other caps?
And put some caps on the input & output of the regulator.
You should really have a current limiting resistor in series with the LED - you're going to damage the LED or damage the output pin wired up as it is.