There are voice shields which I suppose might enable you to achieve something of the sort with an Arduino + amplifier + speaker.
If you want to use a separate MP3 player for playback then you'd need to figure out how the Arduino would control the MP3 player. I suppose you might be thinking about hacking into the MP3 player and replace buttons and switches with relays controlled by the Arduino.
Given that what you're describing is essentially just a different type of MP3 player, is there any chance you could find an existing product which does what you need?