That's always been one of my biggest questions in electronics. What's the difference between high side driver and low side other than the type of transistor used? I always chose based on which transistor I had handy. I would imagine I have chosen wrong more than once.
You need to discern (at least) two different situations here.
It primarily depends on just what sort of device you are switching, and what its supply voltage is.
If the device to be controlled has only two connections to supply power to it and requires more current than the Arduino can comfortably provide, then "low-side" switching is likely to be appropriate. This is particularly relevant if it is operating from a power supply external to the Arduino where the two control wires leaving the control circuit are ground (going to the external power supply) and the low-switched line.
If on the other hand, the power is to be supplied from the Arduino, then it is preferable to use "high-side" switching where either the current is less than 20 mA and the Arduino pin supplies it directly, or a high-side (PNP/ P-channel) switch is used. Again, the two control wires leaving the control circuit are the switched line and ground.
So for example, an indicator LED with a 330 ohm resistor to limit its current to 10 mA could be powered directly either returned to ground or 5V. If external to the immediate housing however, it should return to ground and the resistor be fitted adjacent to the MCU so that any short circuit of the external wiring either to itself or another ground will not cause harm.
If on the other hand, you wish to drive an IR LED at 100 mA and power it from the Arduino 5V (however supplied), you need a control transistor. If it is again, contained in the same housing as the Arduino, then a low side driver would suit as well as high-side but if external, then a high side driver is preferred.
The situation is complicated if the device controlled requires I/O other than its power line. You have to consider what happens when it is switched off. If you control the power on the ground side, then its I/O connection to the Arduino will be pulled high and may feed current back into the Arduino circuitry. Perhaps more to the point, if that device is connected to other devices, some of those are likely to have their negative supply grounded such that the device you thought you had switched off, is still switched on through a ground path indirectly back to the Arduino which is a very undesirable situation.
Of course it should also be mentioned that if you do implement a high-side switch to power that external I/O device, then switching it off will pull the I/O lines to ground, but that is largely a much more manageable situation.
In general, it is overall safer if the principal connection from the Arduino control circuit to any external components including a power supply is the ground.