As stated above, the code you posted is not complete and won’t compile. Please fix that.
I’m gonna guess that ‘set’ and ‘enc_count1’ are global and shared between the ISR and loop().
Make sure you declared them as volatile. Also, the block of code in loop() that accesses the shared variables should be surrounded by a noInterrupts() / interrupts() pair. Then, rearrange loop() so that you do the minimal amount of stuff while interrupts are disabled. Don’t use Serial.print() until interrupts are enabled again:
void loop() {
int tempCount;
unsigned long tempTime;
if (set==true) {
noInterrupts();
tempTime = millis();
tempCount = enc_count1;
set = false;
enc_count1 = 0;
interrupts();
timeold = timenew;
timenew = tempTime;
rpm = some function of tempCount, timeold, timenew;
Serial.println(rpm1);
}
}