The only pins connected between the XBee and the Arduino, via the shield, are the XBees TX and RX pins, which are routed through the switch to either 0 and 1 or 2 and 3.
The holes you refer to are so YOU can connect other XBee pins to other Arduino pins.