That 20 ms is debounce only, after which a button is considered pressed.
But it has to be pressed for 200 ms before any further action is taken - upon which you check for other buttons to be pressed as well, to detect multi-presses.
All in all it's not that trivial. Simply doing a bunch of digitalRead() calls and combining the results and working with that is not going to work well.