Paul sometimes your little comments are enough to get the brain going in the right direction without giving away the answer. This is not one of those times...
Let's recap. You said:
I removed all the delays except in the interupt.
and I said:
There should be absolutely NO delays in an interrupt service routine. None. Nada. Zilch. Zip.
Not that there are any in yours.
So, clearly, there was nothing wrong with your interrupt handler.
I don't understand this, though:
sei(); //Enables interrupts
delay (1000); //Wait 1 second
cli(); //Disable interrupts
Interrupts are not allowed to happen, for ANY interrupts, including the clock ticking, except while in this function. Not a good idea.
You need to rethink how you check the flow/interrupts per unit time. The blink without delay example bears looking at.
You really need to stop disabling interrupts.