If the encoder AB-signals are processed in software which is not running in real time, you might have to look for a hardware interface doing the counting. Another option could be jitter of of the encoder AB-signals, which the software can't catch. You can find hardware interfaces for incremental encoder here: iC-Haus Homepage - product: iC-MD.