You shouldn't have delay calls in your encoder ISRs anyway. I normally poll encoders using either a tick interrupt or a task in a cooperative scheduler. That avoids tying up 2 interrupts and makes it easier to support several encoders. Polling at 1ms intervals is best for avoiding missed transitions, but you can typically get away with 2ms. You can find the code I use at GitHub - dc42/arduino: Reusable modules, drivers and patches for the Arduino platform.
Of course, if you don't have too much else going on (for example, you don't use any library functions that may block such as Serial.print), and there are no delay calls elsewhere in your code, then you can do the polling directly within loop() instead.