The size of a source file has little to do with the size of an object module. The 20kb and 17kb don't add up to create a 37kb object module.
Could I remove parts of those libraries that Cosm doesn't need to make it smaller?
That's the linker's job, and it is really good at it. You don't call a function, the linker doesn't include it. You do, and it does. The only way to stop the linker from including something is to stop calling it.
Or is it possible to store part of the program or library files on the SD card on the board?
No.
Do both the .h and .ccp file become part of the final sketch size?
See above.
so which is more important to focus on?
The code you control.