I am attempting to power up battery powered (4.5V) LED strips and an arduino with the press of one button.
I read that you can have multiple voltage sources sharing a common ground, and I am trying to use that to switch everything on and off.
This circuit was working when before I swapped out a weaker 9V battery (was down to about 5V) with a fresh one. After swapping, the arduino is now powering on, without the switch being engaged.
Your diagram does not show the correct connections of the LED strips.
The red lines to the LED strips and to the Arduino differ by 9V-4.5V = 4.5V, that's sufficient to let current flow even if the switch is off.
I'd power both the Arduino and the strips from the same 4.5V battery, and switch the +4.5V line from the battery. An Arduino will work with 4.5V on Vcc, bypassing the on-board voltage regulator.