Setting non-linear steps from rotary encoder for motor speed control

I would go about this task very differently, and (I think) a lot more simply.

Rather than use a timer to check the encoder I would set up the encoder so that every pulse causes an interrupt. And then in the Interrupt Service Routine (ISR) I would increment or decrement the encoder count, save the value of micros() and set a variable (let's call it newPulse) to true. Actually if pulses come from two encoder pins you may need two ISRs - but they will be almost identical.

In my main code I would be checking if newPulse == true and when it is I would compare the new encoder count with the previous one to know which direction and I would compare the new saved value of micros() with the previous saved value to get the time between pulses. Then you have all the data you need to update other parts of the program.

...R