Brushed DC Motor Speed Control with Incremental Encoder

You are running your duty-cycle adjustment everytime round the loop when the encoder input is high.

You need to detect changes in the encoder pin state, you are not doing so. To detect change you need a variable that records the state of the encoder pin last time round the loop to compare against.