With the classic Arduino IDE, yes. But uninstall capability has been added in the Arduino IDE 2.x Library Manager.
However, as you noticed, Library Manager doesn't recognize a library such as your broken installation of Adafruit BusIO (likely just an empty folder) where there is no metadata (library.properties file). So it would not have offered uninstallation for that library in your case.
Even though the IDE does still support for the "legacy"/"1.0 format" libraries that don't have metadata, it was granted reluctantly at the introduction of the new format and it is likely that improving support for such libraries is not at a high priority for allocation of Arduino's development resources.
Some additional reasons:
- Because there is a bug in the new version of the library.
- Because the library developers made a breaking change to the API in the newer version and your sketch or library relies on the old API. (a prominent example is ArduinoJson 5 vs 6)
- Because your code was tested with the previous version of the library and you don't feel like taking the chance that it does not work correctly with a newer version it has not been thoroughly tested against.