When 0 is 0V, and 1023 is 5V, 1 is pretty close to 0V. Perhaps your environment is simply more electrically noisy.