It uses the static variable old_AB to store the previous and the new state of 2 pins. It does this by shifting its value two bits to the left ( <<= 2 ) and then adding the lower 2 bits of ENC_PORT. (Direct access to the register rather than digitalRead().)
So the lower 4 bits of old_AB contain the previous state and the new state. That makes a number from 0 to 15 which is looked up in this table: (Edit: it is not looked up. It is used as the index.)
static int8_t enc_states[] =
{ // Edit to add more comments:
0, // 00 00 no changes
-1, // 00 01 low, up
1, // 00 10 up, low
0, // 00 11 both changed
1, // 01 00 low, down
0, // 01 01 no changes
0, // 01 10 both changed
-1, // 01 11 up, high
-1, // 10 00 down, low
0, // 10 01 both changed
0, // 10 10 no changes
1, // 10 11 high, up
0, // 11 00 both changed
1, // 11 01 down, high
-1, // 11 10 high, down
0 // 11 11 no changes
};
If the first 2 bits are the same as the last 2 bits, there is no change, so the result is zero.
If more than 1 bit is different between the two (Edit: so that would be both bits), it is an impossible transition, so the result is also zero.
If only one bit has changed, it can tell whether you rotated one step to the left or right, so the result is 1 or -1.