Hi,
sorry that I have to say that, but I think both IC's are not the best choice.
Your CMOS inverter is pretty slow - remember that the PWM is operating with full 16 MHz resolution, so 1 bit change means a length of 62ns, whereas your inverter is way slower than that! Even if that might not be the source of your issues this could make the audio quality worse. I recommend fast ones like SN74HC14 or even SN74AC14. The last one is so powerfull that it is often enough to connect the speaker just to a bunch of paralleled inverters. But for 16-Bit you will need a little modification then (the resistors are then too big).
Second: You are using a class-D amplifier. Have you tried it before using some other active speakers or line-in from other amplifier? The issue is that my circuit is lacking a low pass filter and in combination with Class-D amp this might not be a good idea. So just try to add a capacitor between ground and the point where the 1k and 220k are connected together. Experiment with different values e.g. 1nF, 100nF up to 1uF.