That was the design choice made.
Connect your external grounds together and bring them back to a common point.
A pin can source current to drive an output high, or it can sink current to bring an output low.
The pin is capable of ~20-25mA in either direction with the High and Low voltage degrading as current increases (dropping from a High level or rising from a Low level).
The pins are not current limited and will smoke themselves if you should try to put more than 40mA thru them/