I assume a rotary encoder with switch outputs like in this tutorial. Do you have the pullup resistors installed (no way to tell from the photo). If you enable the internal pullup resistors you won't need add them.
Use pinMode(pin, INPUT_PULLUP).
And the 500 ms delay in every iteration of loop() degrades responsiveness. The millis() for timing guide will help you get rid of delays.