While you could go down the path of pre-compiled files, I would not go there.
IMO, it seems to be a bunch of work and likely can't fully solve the issues,
since how/where it is done and how the files are installed varies by environment and build tools being used so it seems to have risk in that this capability may or may not even be possible for all the desired combination of tools & environments.
And it will likely be a total PITA to maintain for the environments, since you may need to also create special installation tools - which again can vary not only by environment, but also by OS platform being used.
IMO, it is a matter of creating a proper directory structure or if necessary using some automation scripting to create library distribution packages for each desired environment.
In other words can't you create a directory structure that "just works" in all the environments?
If not, I'd suggest that you use some automation scripts to create a distribution package for each supported environment.
That way the master git repository source tree where the source is maintained doesn't have to exactly match the source tree needed for each distribution and each distribution can have its own github repository as needed to simplify installation.
A github repository is needed for any library that you want to be supported by the Arduino IDE library manager.
I use automation scripts often. I use them on my Arduino library packages for things like creating distribution package releases and setting up github tags.
It does things like build all the documentation files, in some cases from Doxygen, in other cases it extracts git tag information an patches header files to insert revision information or http links to documentation or github files and even automatically commits updated files and generates semver compliant tags so that the Arduino IDE library manager can "see" the new release.
In some cases it builds an alternate local source tree and then creates a zip package from that alternate tree that is put up on a server for users to download.
I have one Arduino very large lcd library with MANY sub directories to organize things like header files, fonts, device files, config files, documentation, examples, bitmaps etc...
For this library, the git development tree is not the same as the distribution release tree:
Here is the development source tree:
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
HTML 448 7601 2534 114327
C/C++ Header 234 4853 14129 24685
C++ 21 3526 12039 12713
Arduino Sketch 105 3162 6940 12410
Javascript 108 282 219 4804
CSS 8 548 104 2635
Java 24 484 358 2559
Bourne Shell 5 73 254 186
Perl 1 7 0 105
DOS Batch 3 28 88 50
make 2 10 18 34
-------------------------------------------------------------------------------
SUM: 959 20574 36683 174508
-------------------------------------------------------------------------------
and here is the distribution tree:
-------------------------------------------------------------------------------
Language files blank comment code
-------------------------------------------------------------------------------
HTML 139 402 878 13866
C/C++ Header 113 1900 5383 10657
Arduino Sketch 73 1685 4098 6582
C++ 8 1002 2729 4294
Javascript 88 132 105 2442
CSS 4 309 45 1560
Java 1 35 25 259
Perl 1 7 0 105
make 1 6 12 14
Bourne Shell 1 2 12 3
-------------------------------------------------------------------------------
SUM: 429 5480 13287 39782
-------------------------------------------------------------------------------
--- bill