I have set up 5 relay modules with esps8266 connected to my WIFI and subscribed to different topics on my online mqtt server from which I can control and enjoy home automation.
Now I'm wondering if internet goes down, how can I control all of these modules from one place. would you please share your ideas or point me to a direction.
Given I'm still connected to the same router (all devices connected to) but without internet access
For a simple (and cheap) home MQTT server you can use a 'Raspberry Pi Zero W'.
If you want to do more things on your server then maybe one of the bigger Raspberry Pi's would be better. I use a Pi 3 that runs MQTT, Node-Red and an Influx TICK stack.