Did you try the following test, from that Instructable, to verify correct encoder operation?
A little experiment I did with multimeter :
power the encoder with 2 AA battery, attach multimeter GND cable to GND of the battery , and multimeter RED cable to the C1 or C2 output of the encoder.
As you turn the motor slowly, you can see that the output voltage of C1 and C2 change from 0V to 3V.
If you have 2 multimeter, attach C1 and C2 each to a multimeter, then turn the multimeter slowly you can see the value of multimeter change from 3V-3V, 0V-3V,0V-0V,3V-3V, which is the pulse according the background theory)