Your basic idea is sound. One generic library for displays is, in effect, what device drivers are about on PCs and Macs.
On Windows, for example, a program "writes on the screen" and leaves the low-level decisions about how to exactly do that to the video card device driver.
However we haven't come quite that far on the Arduino, and perhaps we won't because of space limitations.