That would be much more elegant and friendly if it could omit the HID stuff until added by a .begin() statement. My mindset was more in coming at it from a 'hacking' perspective and what would be easiest for an 'advanced' user to modify, while leaving it all simple and unchanged by default.
What I have found as a relatively straightforward hack is in the core files, in main.cpp if I comment out the USB.attach() line, the USB is never initialized and the compiler optimizes away all the USB code even though it was included in the sources. So there's no HID but also no CDC either, and the compiled sketch is about 3kB smaller. There's also about 50bytes of RAM freed up.
Anyhow, thanks for the replies! If I can help somehow I will, but my C / C++ are not great.