I want to turn the power supply to my Arduino on and off using a switch.
Well, then use a switch. If it's just the Arduino on its own, it's only using about 100mA, so any switch rated to switch more than 150mA will work (basically every switch you can buy anywhere).
If you are using a plug pack, turn it off at the wall. If you are powering from USB pull out the plug.
We need more info on how you are supplying the power.
Weedpharma