I believe your problem may be with the encoder's output IC type, read the page "50 | Notes and Tables" in the datasheet ( post # 19). If the cable has 2 unused "complementary" channel wires, what are you doing with them? Maybe they should be pulled to ground with pulldown resistors.