A schematc or wiring diagram are helpful when asking a hardware question. Show all power supplies.
Links to technical data for components are also helpful.
I have a similar project. It works great from a 3.7 LiPo (18750) and a 2A boost converter.
Are all grounds connected?
Is there any change if you don"t use the amp and drive just one speaker from the DF Mini?