Please post your example code that shows the performance problem.
How much interrupts per second do you want to process?
2 interrupts pin can read 2 different encoders.
Pin change interrupts are not to difficult (helped GrayNomad to optimize his lib). Problem is that you get an interrupt and then you need to check all pins to see which (multiple) pins did change. This takes time.
Another solution uses an additional XOR chip on the A and B lines of the encoders
A B XOR
-----------------
0 0 0
0 1 1
1 1 0
1 0 1
0 0 0
By putting the XOR line to the interrupt pin you get all the changes of the encoder.
In the interrupt you need to read the pinA and pinB which are connected to e.g. pin 3 and 4
Reading those 2 pins can be done fast on registerlevel, comparing them with previous value
indicates the direction
Give it a try