What about the overhead of audio decompression? The only way streaming systems can work is by compressing the sound data a lot. Ogg or MP3 decoding isn't exactly a breeze. On that subject, what is your level of programming expertise?
I would pick something like an RPi even if it is overkill, just because things like Ogg are freely available for it, and no worries about processor speed. IIRC, the audio hardware is available too. Basically you are just creating an alternate computer network. You could do it with surplus laptops or things like that, too.