In the library the encoder debouncing is based on the encoders own hardware "state machine", IE is the new state valid based on the last state, as there are known conditions of A and B that are not valid.
It's a pretty common way of doing it. However, in your case, if the encoder is very noisy, you could add hardware debouncing as well to try and further prevent invalid states from registering.
So there are a few tradeoffs in the library, it was originally built mainly for use by menu libraries but has been used in a very wide range of builds now. Here are a few ideas.
If you're not trying to run in low power mode then disable the interrupt support by changing switches.init(..) from SWITCHES_POLL_KEYS_ONLY to SWITCHES_POLL_EVERYTHING, this means it will always poll for rotary encoder changes. This removes the possibility of state changes between the interrupt triggering and task manager servicing it. See the comment above the switches init call in the example.
You can also slow down or disable the acceleration by providing the default parameter to the HardwareRotaryEncoder constructor. For example:
auto hwEncoder = new HardwareRotaryEncoder(encoderAPin, encoderBPin, onEncoderChange, HWACCEL_NONE); // or HWACCEL_SLOWER, HWACCEL_REGULAR
At the end of the day, if someone turns the encoder exceptionally quickly it may be difficult to debounce fully in software. The only thing I can think of that the library doesn't do now, is to ignore direction changes for a few internal ticks, but that comes with some overheads, memory and processor.
I don't like changing this class much, as it's used by very many users, and we can see what happened last time it was changed, the signature of a method was silently broken.