Sketch uses 20,922 bytes (64%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,585 bytes (77%) of dynamic memory, leaving 463 bytes for local variables. Maximum is 2,048 bytes.
Low memory available, stability problems may occur.
It is said that the IDE (or compiler) only compiles or links what is needed. A simple test seems to prove that but I'm not 100% sure if I did it correctly. I don't have the GFX library installed so can't test your scenario.
I created a sketch and added two files (simulating a library); this 'library' contained two functions and in the sketch I only used one of them. Removing the non-used function from the 'library' did not make a difference in the memory (flash and ram) usage.
You can easily try it yourself by removing a function (that you don't need) from the library's cpp file and check the result.
Is there any way I can trim down the code to remove anything thats not required.
The library needs a buffer to hold one bit per pixel. If you have a 128 x 64 pixel display, it needs 1024 bytes of memory to hold that data. There is NO way that you can reduce that.