It's not good to drain the rechargeable batteries until the router shuts off, that will be well below the recommended discharge.
The boost module will keep boosting until there is not enough current or voltage to do so, regardless of the battery pack's state of charge.
Go with your assumption that the router will run for 2 hours and see how many cycles you get before the time your load runs becomes dramatically lower or your batteries won't charge properly.
Be aware that as the voltage of the batteries drops the modules efficiency gets worse.
For better efficiency raise the voltage of your battery pack.
2 x 9.6v battery packs in parallel a 4aH total capacity will give you around 2aH of useable current or 19.2W.
That you could safely say will run your router for 2+ hours.
As for adding another router, it might "work" for you, but with your current setup it surely will not be "properly working".