Could I ask you to look at your code again?
First, I think I found a bug
The value in 'command' is not cleared if there is no serial character available. This causes the command to be continuously repeated. I added an else which clears it.
OK now the next bit is a bit fuzzy because I'm a newbie at arduino.
This is about some things that are not working in the program.
Firstly, I ran the program and the count and error count prints are always zero.
I only had a MEGA2560 at the time and after a bit of research I found the timers are a bit different. Since I had a UNO already on order I put the project on the back burner.
Now that I have a UNO. I tried again and found the same problem that there are only zero counts on the serial monitor.
Ok so here is what I know:-
The quadrature signal at digital 9 & 10 is present and is 5kHz period as measured on my scope. Is 5kHz correct?
The serial monitor shows zero for count and zero for errors.
Forcing errorCount to increment in the ISR also shows a zero count in the serial.
Commenting out the errorCount=0 doesn't change anything
So what I'm wondering is, does it really ever enter the ISR?
The code you provided looks like a snippet from some larger program and it contains stuff that I don't understand.
For instance the serial monitor section looks like it was originally intended for an external 'real' encoder because why would you need the error counting in a quadrature generator?
The external interrupts are on digital 2 and 3 aren't they? So do I need to physically link them to digital 9 & 10 to get the ISR to run?
Now here comes the question that I don't know how to find elsewhere and I hope you can educate me.
Can I generate pin change interrupts from digital 9 & 10 when they are outputs? Is it covered in the datasheet?