Sounds as reasonable as it gets. I suppose ideal would be a USB hub chip that supported port disable, but I don't see any of those.
Your first design sounds like you wanted a crossbar switch. There are integrated circuits for this, but sadly since you end up with a lot of pins they are all in packages that are tricky to play with like ball grid arrays/ In any event, you don't need a crossbar, it takes a lot of switches and allows each input to be connected to many outputs at the same time, very bad if you think of your USB devices as the inputs.
What you really want is a multiplexor that can connect each device to one of several outputs. Unfortunately I'm not seeing a 4:1 mux in a DIP package with low enough resistance to not screw up your USB. But for very low power devices you might get by with a pair of MAX4618 (2 bit 4:1 muxes with a 10ohm Ron) for each USB device. This probably doesn't work for USB 2.0 HiSpeed devices, it looks like it starts degrading about 20MHz.
Alternatively, you could wire each device to all hubs simultaneously but insert a switch to interrupt the connection. If you have some soldering skill or know someone who does you can get little adapter boards to mount the tiny packages on and then break out the signals to something large enough to get on a breadboard. That said, you might look at something like a Fairchild FSUSB31 which is tailor made for making and breaking a USB connection and in stock at digikey. Somewhat less ideal would be something like a Maxim MAX4677 which is 4 bits of SPST switch in a 16 pin DIP so you can easily handle it. The downside is that it isn't fast enough for USB 2.0 HiSpeed.
And I wondered what would happen if you left the +5 and grounds connected all the time and only switched the data lines. I thought that would work, but you would not be in compliance with the USB spec. It demands that your pull up or pull down resistors that identify your speed only get driven when the host supplies the +5v.