Large Error Instability

One possibility is that your ISR function for handling encoder interrupts is too long, complex, or too slow to keep up with the interrupt rate. Without posting your code it's very difficult for anyone to help with your symptoms.

Lefty