Making Library's that use direct port manipulation?

Yes you can, using non-typed templates is one way.
In fact if you used templates, you could simply pass the pin number to fat16libs DigitalIO lib which does the pin translation for you. Fast digital I/O, software I2C, and software SPI libraries - Libraries - Arduino Forum