Pin numbers only have relevance within the Arduino environment -- whether you use the IDE or not is, as you say, of no concern. However, if your goal is to support as many platforms as possible (both Arduino and general AVR), things like "D0" cannot be depended on. That's one issue.
The second is this: Imagine you're trying to write a driver for a new LCD chipset that uses an 8-bit data bus. Do you really want to pass (D0, D1, D2, D3, D4, D5, D6, D7) to the init function? You can gain a ton of efficiency by using atomic 8-bit operations on a whole PORT instead. In some cases I've seen, when a developer writes a library like this, they create it with the intention of running on an ATmega 328, and tell you which pins will be consumed. If you hope to use that library on a 1284, well... prepare to track down all the hardware-specific parts and re-write them yourself.
I'm looking for a way to support any current or future MCU (within reason) without having to republish the library with chip-specific layouts when they change, or have the user go and modify library sources because they want to use PORTD instead of B. Unfortunately, I'm just not aware of any good method to allow a library user to specify a port or pin in a portable way, since all the standard defines are intended to ACCESS the port, not provide a way to pass its hardware address through function calls. I'm hoping this is achievable in some way I haven't thought of yet, though I realize it just may not be.
Make sense now?