Can I increase the resistance of my linear potentiometer?

If your readings are full scale (ie zero mm reads 0 (ish) and 50mm reads 1020 (or so) then your error is 0.2/50 * 1024 = 4 LSB which is to be expected.
Likely the best way to reduce the error is to average multiple readings as explained here