why do the basic microprossessor boards from all vendors that I've seen not utilize headers that have pins sticking out the bottom so that additional shields can be plugged in under the microprocessor board?
You don't want to plug an LCD or a LED matrix shield "under the microprocessor board", do you?
Why are the headers often shorter (less tall) than the some of the components on the shields?
That's rarely the case. The highest component I saw was the Ethernet connector. You would need to adapt your shield to fit those boards (or pick one that fits).