I'm powering the Arduino from a 5V bench supply and then stepping down to 3.3v for the DFPlayer with a AMS111733 regulator because the datasheet said the preferred working voltage was 4.2v rather than 5v. To be fair I'm not sure whether I was getting the pop or not. Using the amplifier so destabilised the module that I gave up on it very quickly. I was also getting a loud 10Hz (roughly) ticking sound through the speaker while it was initialising the SD card which went when I took output from the DACs.