The SPKR 1/2 is not a ground-referenced output.
You should use the DAC L/R outputs for this (not the SPKR outputs).
Also, when powered by USB, "5V" is from USB.
When powered by your battery, "5V" is from the Nano's wee, and likely inadequate, 5V regulator.
Those tiny speakers can be handled by the player itself, that amp is adding nothing.