Rotary Encoder Drift??

I would also remove the calls to digitalWrite() in your interrupt handler. You can't trigger another interrupt until the first one finishes, and those calls probably take much longer than the rest of the code in the handler.