Common ground, encouraged. 5V regulator outputs, do not cross-connect. Regulators are not designed to compete, nor co-operatively share loads. You may get away with it, or you may junk one or more Arduinos in short order. Feeling lucky?
Furthermore, more information about your loads is needed, but:
- the 5v output should not be used to power any form of motor. Logic 5V outputs to a motor driver, yes, but provide the motor driver with it's own supply.
buttons, switches and pots feeding Arduino inputs, all your Arduinos should be capable of powering their own share of such devices.
However, don't run 5V from Arduino A to any digital inputs of Arduino B. Because, when Arduino B is powered down for whatever reason, the input protection circuitry (essentially, diodes) internal to the input of the Arduino become forward biased, forming a direct path to ground for the available current of the regulator on Arduino A; damage to Arduino B will ensue.