The code looks good, infact, non-type templates is exactly how I have implemented my library. I'm working on the HAL now to allow it to generate port mapping for any combination of pins ( optimising related pins ).
How many overhead bytes does the pin_map_t arrays produce?