One full rotation is outputting 3 counts and it should be outputting 20 counts. I tried the following code with a rotary encoder and it was giving me the right cpr. I'm not sure why this magnetic encoder is outputting the wrong value.
As others have indicated, you are not reading all the available quadrature transitions. Your code is reading only one of the four transitions available. With a 20 cpr encoder, you should be seeing 5 counts/rev with your code.
There is a very similar encoder with 12 cpr, and that would give 3 counts with your code. Are you certain that you have the 20 cpr encoder?