Citius, Altius, Fortius is the Olympic motto in Latin and translates to Swifter, Higher, Stronger. This is deeply embedded in all of us and possibly the main reason why we are the most successful species on the planet and not something we should abandon easily. For a general purpose computer such as a PC, faster does indeed appear to be a winning strategy.
The success of the Arduino however has come from users recognizing different strengths (other than speed) and successfully applying those in their projects. The Arduino is good at interfacing with the physical world, reading sensors, adding computational logic and controlling output. An Arduino can be embedded in devices we build by itself or it can interface and add functionality to devices that excel in other areas.
Many refer to the Atmega in comparison to the Duemillanove as the “better board”. In my mind however it is just a “different board” appropriate for projects with different requirements. If your project does not require the additional IO or memory it is just a waste of money, space and energy. Likewise, if you need a fast graphics controller, I'm sure you will easily find a different controller (or even a microcontroller other than the AtMega) that is a zillion times faster and more apt for such a task.
Trying to bend and twist the Arduino to fit any requirements may sometimes have an academic value, but less so for practical applications.