I'll work on a wiring diagram but I believe all the grounds are correctly connected since it works with the power from the USB port.
I removed the amp and used just one speaker on the mini player and it works correctly.
I'm guessing that the batteries can't supply sufficient amps to run all thee components or more likely that the 5v pin on the Nano can't supply the required amps?