How do I customise the board manager based libraries?

I know how to hack the libraries but I want to do it properly, do it once, and not have to worry about arduino undoing the changes on me at some point in the future.

I'm specifically trying to work with the ESP32 and ESP8266 libraries.

Obviously both libraries (and the default library) contain more boards types than someone is reasonably ever going to own so my first requirement is to filter them out...

The next requirement involves the need to define pin mappings for an ESP8266 board which doesn't match any of the existing boards. I know it is easy to take an existing board definition and modify it but as per my first sentence I don't want an update via board manager update to result in my changes being wiped.

Unfortunately ESP8266 doesn't appear to be manually installable (which would solve everything) and attempting to do so results in errors like...

No valid hardware definitions found in folder ...

Any help in resolving these issues would be much appreciated.

m12lrpv:
I know how to hack the libraries but I want to do it properly, do it once, and not have to worry about arduino undoing the changes on me at some point in the future.

I’m specifically trying to work with the ESP32 and ESP8266 libraries.

Obviously both libraries (and the default library) contain more boards types than someone is reasonably ever going to own so my first requirement is to filter them out…

Unfortunately there is not a great way to accomplish this that meets your requirement of not being affected by updates.

In order to assist with user modifications of boards definitions. A feature has been added to the Arduino IDE that lets you create a file named boards.local.txt in the same folder as the boards.txt of the hardware package you want to modify. The definitions in boards.local.txt override the ones in boards.txt. The benefit of this is that the user doesn’t need to modify boards.txt. Unfortunately, the boards.local.txt file will be deleted every time you update to a new version of the hardware package. You can store a copy of it somewhere safe and then copy it over to the hardware package folder after the update. You will need to restart the Arduino IDE for your boards.local.txt to take effect.

There is an undocumented feature that allows you to hide boards entries from the Tools > Board menu by adding a property named hide to that board. For example, if you wanted to hide the ESP8266 hardware package’s “Generic ESP8266 Module” board from the menu, you would create a file named boards.local.txt in the same folder as the ESP8266 package’s boards.txt and add this line to your boards.local.txt:

generic.hide=

where “generic” is the board ID from the boards.txt definition of that board:

generic.name=Generic ESP8266 Module

m12lrpv:
The next requirement involves the need to define pin mappings for an ESP8266 board which doesn’t match any of the existing boards. I know it is easy to take an existing board definition and modify it but as per my first sentence I don’t want an update via board manager update to result in my changes being wiped.

In this case, the situation is better. The Arduino IDE allows hardware packages to reference components of other hardware packages. So you can create a custom hardware package for your ESP8266 board, which only consists of a boards.txt entry and a variant file, then reference all the other necessary components from the esp8266 vendor. You can find details on this here:

So if you want to reference the esp8266 hardware package’s core and you chose the board ID “myboard”, you would put this in your boards.txt:

myboard.build.core=esp8266:esp8266

The first “esp8266” being the vendor name (as determined by the vendor folder name in the ESP8266 hardware package installation) and the second “esp8266” being the core name (as determined by the core folder name in the ESP8266 hardware package):

Arduino15
|_packages
|_esp8266 <— vendor name
|_hardware
|_esp8266
|_2.4.2
|_cores
|_esp8266 <— core name

m12lrpv:
Unfortunately ESP8266 doesn’t appear to be manually installable (which would solve everything)

Yes, you can do that, following these instructions:
https://arduino-esp8266.readthedocs.io/en/latest/installing.html#using-git-version
However, I don’t see how that solves anything. Sure, you can modify the manually installed hardware package, but you can do the same with the Boards Manager installed package just as easily. The only difference is that Boards Manager makes it easier to keep the hardware package updated. If you don’t want to update it, you’re not forced to. So that is no disadvantage.

Thanks for all of that. That's great info and very much appreciated.

I knew there would be some way to achieve what I needed.

I'll make the customisations you've suggested and if updates overwriting them becomes a problem I'll write a launcher to ensure they're in place each startup.