Posting code fragments will might get you guesses, not answers. You cannot test compile or test fragments.
Here's my guess.
void count1()//ISR
{
//set=true;
static int8_t lookup_table[] = {0, -1, 1, 0, 1, 0, 0, -1, -1, 0, 0, 1, 0, 1, -1, 0};
static uint8_t enc_val1 = 0;
enc_val1 = enc_val1 << 2;
enc_val1 = enc_val1 | (PIND & 0b11) ;
enc_count1 = enc_count1 + lookup_table[enc_val1 & 0b1111];
}