Ghost Pin output

I am using UNO R3 for the first time and I got 2 boards. I first tried a very easy program as below

int testPin = 2;
int ledPin = 4;
void setup() {

pinMode(testPin,INPUT);
pinMode(ledPin,OUTPUT);
}
void loop() {

if(digitalRead(testPin) == 1){
digitalWrite(ledPin,1);
}
else{
digitalWrite(ledPin,0);
}
delay(100);
}

I connected a LED's long leg to 4, and short leg to ground.
The problem is that when I plugged a wire to 2pin without connecting the other end to any power source, sometime the LED would light for a while. Also sometimes when I swing the other end of the wire, it mostly would light up the LED too. However, when I connect it to the ground the LED must switch off.

It happens to both of my boards, I don't know what is happening but it seems that it is a very serious problem...

It's not a serious problem, it's just a floating high impedance input.

I connected a LED's long leg to 4, and short leg to ground.

That's a serious problem. You need a series current-limiting resistor.

Thank you very much for your reminder! I would be careful next time!!!

Well, would this mistake cause permanent damage to the board already?

The floating pin won't cause damage, but the missing resistor may have.