I've always considered RPi to be an overkill for this task. However @Paul_B you confirmed my fear that Arduino might not be sufficient for the task.
As for @SteveMann's proposal to just use Amazon Alexa - it's true, my idea shares a lot of functionality with the Echo devices, however I want something that is wired (for number of reasons I prefer not to use WiFi for this) and at the same time I want to have the freedom for custom control both in regard to hardware and software. Additional reason I considered Arduino for this was the price - given that I need at the least 8 of those with the option to further add at least 4 more, the total price of this gets on the steeper side if going with Echos.