Initially I thought about doing this by manually driving the speakers of different pins, but I think that a multi-channel amp or similar that can be controlled
You can't "drive speakers" directly. You need an amplifier.
Where's the sound coming from? i.e., Are you planning on generating sounds with the Arduino, or will you be playing a 5.1 channel DVD, or something like that?
You can control digital potentiometers (digital volume controls) with the Arduino. That's probably the easiest way to pan sound around.
Some limitations...
-
The Arduino doesn't have a true digital-to-analog converter. You can add an audio shield, but I'm not sure about multi-channel sound... You'd need multiple audio shields and possibly multiple Arduinos. If you just wanted to make some simple "tones", then you have plenty of I/O ports for plenty of channels.
-
The Arduino's analog-to-digital converter is only 10-bits. Possibly good enough, but CDs are 16-bits.
-
The Arduino doesn't have a lot of memory to store audio files (although you can add it).
-
The Arduino doesn't have a lot of processing power for DSP (digital signal processing).
...For those reasons I wouldn't use the Arduino to store or process the audio, but it's fine for controlling digital pots or analog switches, etc., and otherwise controlling the "flow" of the audio.