I've just done this, using (effectively) the Adafruit MCP4922 Wave Shield, and the MAX9814 Electret Mic breakout.
XBee is only 32 kbit/s user data throughput. That is the limiting factor. Nothing about the AVR ATmega is going to limit this application. I'm getting about 24 kbit/s reliably out of the XBee, which is enough for (quite) low quality voice. But, it can be understood and it does get the message through.
It makes a very good project to do.
How to build an encrypted digital walkie talkie.