With opamps and digital to analog converters and other analog circuits, there's a specification called PSRR, which stands for Power Supply Rejection Ratio.
For example, a DAC or opamp might have PSRR of 60 dB (values from 50 to 90 dB are common). Typically it's specified at a certain frequency, and for better devices a graph is sometimes given showing how it varies by frequency.
So if you've got a PSRR of 60 dB and you've got 50 mV of noise on your power supply, it will be attenuated by 60 dB (which is a reduction of 1000 times), so you'll get 50 uV of noise on the output. Obviously you want your power supply to have as little noise as practical, and you want as much PSRR as you can get, so the output doesn't get noise from the power.
With filtered PWM, you get the worst case in both regards. There no PSRR at all. Or maybe 6 dB if you want to play with specs and assume on average your signal will be 50% PWM duty cycle. But the point is any noise on your power supply couples directly to your output circuit, because the PWM is just turning on a transistor inside the chip which shorts the two to drive the pin high. Worse yet, that's the noisy digital power supply, not a nice analog supply which has lots of filtering and powers only analog circuits that draw (appoximately) constant current. The digital supply powers lots of other logic circuits, which are switching rapidly. Even with good decoupling capacitors, you can expect quite a lot more noise on your digital supply... which of course couples right to the output signal, since PWM has no PSRR at all.
Of course, your problem might not be related to PSRR. It might be a ground loop. There too, PWM is pretty terrible, since it produces output relative to the same digital ground that's running lots of other stuff which switches and draws pulsing currents, which of course must flow through digital ground wires. If any of those currents induce voltages relative to your amplifier, you'll heard them added to your signal.
The bottom line is PWM is pretty terrible for quality audio. You can take steps to lessen the issues, but ultimately it's very hard or pretty much impossible to get a signal that anyone who really cares about sound quality would consider "clean".