Also keep in mind that rotary encoders can be kinda jittery. I recommend a hardware debounce, which might be a little more complicated using a resistive network. Maybe not, but it would sure help to see the waveforms on a scope first. Do you have one?
On a related note... The nice thing about the interrupt / digital read is that the signal is either above the comparator or not. On an analog read, you could potentially be sampling somewhere along the LOW / HIGH transition. So, if your debounce slows the transitions much, you have that statistic to contend with as well.
Even with a human turning the knob, quadrature encoders are really chatty. You may need to change the ADC resolution to get faster samples (both to catch the pulse and to finish reading quickly enough to be available to catch the next one), which will of course make your readings less precise.
In other words, it's entirely possible, but it will likely complicate things. IMHO, encoders are complicated enough on "slow" GP microcontrollers as it is. You could always look into a dedicated driver IC that speaks serial, or get a knob with such a thing built in. Just a thought.