Arduino Game Controller Hub

Okay, I have been thinking about a project for a very long time now, and I'm about to start actually building it now, but I need to plan it out a bit more first. I'd like to avoid as much trial and error as possible (even though there will be lots of it regardless) so I hope you guys can help with that.

I have a lot of game consoles. Like almost all of them. Newer consoles have wireless controllers which is nice, but a lot of retro consoles have very short cables for their controllers which can be a pain sometimes. Sure, I could use extension cables, but it will just be too many wires all over the place. I want a tidy solution.
So my first thought was to mod each controller to be wireless, either via Bluetooth or RF, but that can be very tricky depending on the controller, and I don't want to use any third party controllers.

My next thought was to build a hub of controller ports that I place on (or build into) the table in front of my couch. Functionally this would work like I just had extension cables for each controller port on all my consoles, have the wires go to my hub and then I can plug in my controllers there without worrying about cables reaching. Now take that concept and remove the cables between the hub and the consoles. That's what I want to build.

I'm thinking RF communication is the fastest and most reliable here to avoid as much input lag as possible. But if I have too many RF tranceivers it's just going to be chaos and they are going to disrupt each other. So what I'm thinking is this:

  1. Controller port in my hub, connects to Arduino
  2. Arduino (Nano might be enough) to read the input data from controller, then send it forward to the next Arduino
  3. An Arduino MEGA (perhaps) to receive the data on all controllers, serialize this data and send it via RF to a second MEGA on the console side
  4. The Mega receives the data and breaks it down, checks which console it's going to, sends the respective data to each controller port on that console.
  5. The whole process is reversed with data sent from consoles to controllers
  6. Repeat ad infinium

This is what I have theorized will work at least, but practically I have no idea. Since only one console will be used at a time (it's all connected to the same TV) a maximum of 4 controllers will be active at the same time (for N64 for example). But I have never attempted this much intercommunication between Arduinos before, so it might not work at all. On the Nanos I can use the TX/RX pins to send and receive the data from the MEGA, but the MEGA will have to use SoftwareSerial to be able to read this many inputs and this might be too slow? I'm not sure. I am expecting some input lag regardless of how well I build it all, but it's more than 0.5 seconds it might be too much.

So, is this even doable? Is my thinking completely off? If it does work, is there any way to increase it's efficiancy? Any and all advice would be helpful!