encoder troubles

Then you need to add a suitable low-pass filter. analogRead() cannot read PWM signals, only true analog voltages.

OR

measure the duty-cycle of the signal using the pulseIn() function.