Hi I am looking for advice on web scraping via ESP32. Have looked a little at Beautiful Soup.
I am trying to build some resilience into my solar and battery powered remote security wifi camera which uses a 4G modem for internet connectivity for remote access and cloud storage of videos.
Whilst my power generation and consumption budgeting are fine for the overall system, I want to be able to monitor the battery level of the rechargeable battery inside the 4G modem through screen scraping by an Arduino ESP32 device (Wemos D1 R32 is preferred). I then want to get that info remotely - perhaps by loging into the Wemos via a virtual port (the DLINK doesn't have an internet-facing admin interface).
I have a DLINK DWR-933 4G modem which has the usual router web server for admin. It has a login and popup screens containing different information and settings.
I have a login/password page to get past first, then I can periodically check the battery level (and megabytes used).
I have checked the web page in Chrome developer view and can see the settings I need to get.
If I develop a solution that is essentially building the internet facing web page that Dlink hasn't provided, then I will probably have to use a DDNS updater in the solution.
If I develop a solution that transmits the info (eg to a cloud service) directly, then that would be OK.
A further complication/interest is in switching from Wemos D1 to an ESP32 relay board and disrupting the solar charging of the battery-backed camera and battery-backed 4G modem to drive power towards different devices if one is nearly out of power. This would add value to the solution allowing more remote interaction and control of the overall solution.
I live about 1 hour away from the camera installation. I don't have any other means of installing internet access except via mobile telecoms. I don't have any other electricity at the site.