Simplify the reading of the encoder without losing too much. You don't need to have two IRQ's.
The encoder below does measure BOTH direction and the position by comparing the state of both pins with each other.
For position the code uses a signed long so one has a distance/direction indication since start.
e.g. if pos = -1000, we know the sytem did more CCW as CW since start.
This encoder is very small and will interfere minimal with your servo, give it a try..
(code compiles but not tested)
#define encoder0PinA 2
#define encoder0PinB 3
volatile long pos = 0; // long can hold far more pulses than int; signed to know direction since start.
volatile boolean CW = true; // direction if false it is CCW
// SERIAL COM
// CONFIG IRQ
attachInterrupt(0, doEncoder, RISING); // CHANGE will work too but gives you twice the amount of irq's! Do you need that much?
// OUTPUT ENCODER INFO
Serial.print("direction 0=CCW 1=CW: ");
delay(250); // simulate other activities.
// determine direction
CW = (digitalRead(encoder0PinB) != digitalRead(encoder0PinA));
// update position
if (CW) pos++;