Ethernet-based intercom setup?

I posted this elsewhere on this forum, but I'm reposting this here for better vsibility.

I am trying to design a sort of ethernet-based intercom-type setup, where there will be a pair of speakers & a mic in each room. The mic would get converted into a digital signal and streamed to a central pc. The PC in turn, would stream back audio to the device, which would convert it to analog audio to playback on the speakers. I envision the arduino setup would be like a "dumb hub"--it receives input, and does the AD/DA conversion (and any codec work), and then it pipes it out the opposite end.

Does anyone have any ideas/suggestions on getting the necessary hardware to set this up? I've searched around for hifi audio processing shields, and the Rogue Robotics rMP3 shield was the best one I could find. Perhaps it can be modified to fit my purposes?