How about varying the order the buttons are read? then one controller doesn't have a permanent advantage.
If all that happens in the main loop is that the inputs are looked at then there is no permanent advantage for any button. This is because the time taken to go from the last button in the list to the first is little more than from the first to the second. As the bulk of the time is spent in the digitalRead function all buttons have an equal chance.
Do an experiment, wire all the inputs to the same button, a permanent tie, and see how many times the first button on the list is called.
It would be nice to keep the order of who pushed
This means you have to keep looking at the buttons even when one has been pressed.