There's a few things you can do to reduce the interference.
First, you've drawn the power lines as branching off to each device. Give them each their own wire all the way back to the power supply. This is called "star" wiring because it all radiates out from a central point.
You can add capacitors across the pins of each device. Big capacitors absorb low frequencies and small capacitors absorb high frequencies. Sometimes you will need one of each size. Buy a capacitor assortment that has values between 100uF and 0.1uF. Experiment with putting those different values on either the motor, speaker or both.
You can add inductors too. Simple clamp-on ferrites like you see on USB power cables can do the trick. You can get all sorts of different inductors. The neatest ones are integrated into the socket but I'm guessing that desoldering the speaker PCB and putting a new socket on is going to be more expensive and time-consuming than getting a new 5V supply.