There seems to be a lot of assumptions in Arduino-land. I looked through all the code the other day for giggles. SOoo much dependence on 8/16MHz clocks, for one. I bet libraries exacerbate that problem.
Additionally, many functions (in order to make them work uniformly across chips even in the same family) are littered with:
#ifdef __SOME_ARCH__
// Do something low-level
#elif __SOME_OTHER_ARCH__
// Do something another way
# elif __I_KNOW__
#error "I don't know your chip well enough, sorry."
#endif
So there's that stuff you have to contend with, then the fact that a great deal of the capabilities of more advanced chips will never be exposed, due to the lack of corresponding framework. It's lowest-common-denominator indeed.
But, on the bright side, 90% of the code, once a few key functions and includes are rewritten, will just compile. And, you can always take advantage of the other code via direct AVR calls. I'd say this is more of a way to migrate off Arduino code than to make the bigger,better,faster chips into Arduinos, but that's just one way of looking at it.