LEDs losig brithness at a different rate, am I damaging them somehow?

Are you talking about Johnson noise?