The Arduino doesn't have a DAC (digital-to-analog converter) so there's no true analog output. The solution is an add-on [u]audio board[/u]. The Arduino simply acts as a controller to choose the file to play, start/stop the sound, adjust the volume, etc. The audio board (AKA "shield") has memory, a DAC, an audio clock and everything else needed.
I imagine that I would need an amplifier, speaker
Yes, or you can use regular "powered" computer speakers.