As you discovered, the URLs always follow the same format:
https://downloads.arduino.cc/libraries/<Git host>/<Git host organization>/<library name>-<version>.zip
So if you happen to know all those things (<Git host> is github.com for 99.9% of libraries, even though other major hosts are supported), then you know the URL.
If you don't know all of it, you can hunt for an archived version of the JSON file that contains the library. That file is occasionally archived by the Internet Archive Wayback Machine:
https://web.archive.org/web/20220000000000*/http://downloads.arduino.cc/libraries/library_index.json
Note that the JSON file is very large and loading pages on Wayback Machine is very slow, so you must be patient and wait for the full JSON file to load.
You will find the download URL for each library release in the libraries[*].url field of the file.
Wayback Machine automatically converts all URLs to the URL of the archived page on Wayback Machine (e.g., https://web.archive.org/web/20220829173445/https://downloads.arduino.cc/libraries/github.com/bcmi-labs/Arduino_TensorFlowLite-1.15.0-ALPHA.zip), but those ZIP files are not actually archived (Wayback Machine generally only archives web pages), so you must remove that web.archive.org prefix (e.g., https://web.archive.org/web/20220829173445/) from the URL and download the file directly from Arduino's server.
Note that there is no guarantee these files will continue to be available from Arduino's server. I was actually quite surprised to find the file was still there.