To me it sounds like this is something that should be done entierly in software that would run on a PC or MAC (and maybe even on a smart phone).
Writing an application for a computer isn't nearly as easy as writing an Arduino sketch, but you wouldn't have to build any hardware, and there would be no hardware reliability issues. And once you have one copy, you can make hundreds or thousands of copies at zero additional cost!
Is there a simple IC audio mixer that I could use for this purpose?
Something like [u]this[/u]? Or [u]this[/u]? And, maybe a multi-output [u]headphone amp[/u]?
I assume just wiring all of the outputs to a single speaker would not work.
Correct! The general rule is that you should never connect outputs together. However, it is generally OK to connect multiple inputs together such as connecting several amplifiers to the output of a CD player. (You can have trouble if you connect multiple speaker or headphone "inputs" to one amplifier, because of impedance loading.)