Problem with RPM Photo Interrupter/Motor Speed

I think you need to save the value of counts to another working variable and reset the value something like this

latestCounts = counts;
counts = 0;
prevTime = millis();
rpm = (60 / N) * 1000 / (millis() - timeold) * latestCounts;
// etc

That will ensure that the rest of the code interferes as little as possible with the counting.

However I suggest you use a different approach - measure the time for each revolution using micros().

This code is derived from a project where I control the speed of a small DC motor. It is not complete but it should give you the idea.

volatile unsigned long isrMicros;
unsigned long latestIsrMicros;
unsigned long previousIsrMicros;
volatile boolean newISR = false;


void loop() {
   if (newISR == true) {
     previousIsrMicros = latestIsrMicros; // save the old value
     noInterrupts(); // pause interrupts while we get the new value
        latestIsrMicros = isrMicros;
        newISR = false;
     interrupts();
     microsThisRev = latestIsrMicros - previousIsrMicos;
}

void myISR() {
   isrMicros = micros();
   newISR = true;
}

...R