They don't usually just die, they're KILLED. 
From the sound of your description, when you disconnect the power supply, you're drawing current backwards through the regulator's internal diode to supply the LED with (just under) 5V via the Vin pin. That's a big no-no. It will kill the regulator.
You could disconnect the LED before connecting to the computer if the power supply isn't connected, or you could add a diode as you say, or you could supply power via the DC input jack, (with the LED connected before the jack), so that the onboard diode stops reverse current flow. Edit: It's a Nano, not an UNO. I missed that.