<SOLVED> ATmega644 does not catch the interupt

Of course, but they are they same. Assuming that you mean the ISR function that works and the one that doesn't...

This one works (Arduino Uno)

// Rotary encoder interrupt service routine
volatile boolean rotating=false;

void rotEncoder(){
	rotating=true; // If motion is detected in the rotary encoder, set the flag to true
}

This one does not (ATmega644).

volatile boolean rotating=false;
void rotEncoder()
{
	rotating=true; // If motion is detected in the rotary encoder, set the flag to true
}