Yes that is my point, the hardware abstraction can take place under the hood like it is done now and most users will be non the wiser.
Sadly, the Arduino world is filled with sketches and libraries that directly manipulate the timers and ports.
While pinMode, digitalWrite and digitalRead do provide a nice hardware abstraction, lots of authors are dissatisfied with their sluggish performance. Several libraries, like Capsense, Ping, OneWire and others legitimately need the performance and can't work well with the 50-some cycles taken by digitalWrite. But in many other cases I've seen, authors are simply familiar with directly manipulating the ports and do so, even if their code spends much of its time in delay().
Even with the Arduino functions are used, a very AVR trick is used to activate the pullup resistors. All ARM chips, and even AVR XMEGA, use a separate register to control the pullups, rather than repurposing the output register when the pin is using input mode. Arduino 1.0.1 will finally provide INPUT_PULLUP with pinMode(), but there is already a vast amount of code written which depends on the AVR register behavior.
When it comes to timers, Arduino provides nothing other than polling with millis() or micros(). A good portion of all Arduino libraries directly manipulate one or more hardware timer, with lots of AVR timer register assumptions built in. If you look at ChipKit and Maple, very few libraries are compatible. Usually it's timers.
Arduino has also been long missing a number of other important features. In their absence, authors have resorted to direct hardware access. For example, I recently worked on porting (only to Teensy with non-UART "Serial", not a non-AVR chip) a sketch that allowed a specific software package to control several stepper motors. It was based on a big polling loop that depending on non-blocking access to Serial and the motors. Of course, Arduino's Stepper library is a fully blocking API (only 1 stepper can move at a time). Until very recently, transmitting on Serial also blocked if more than 2 bytes were sent. So the author wrote his own stepper control code, using direct port manipulation, and wrote all the output to a buffer, with direct access to the UART registers to incrementally transmit the buffer contents only when the UART was ready for another byte. Had Arduino offered something like Serial.available() to know if writing would block, that sketch probably wouldn't have directly hit the UART and might be portable to a future ARM-based board. But like so many Arduino application, the gaps in Arduino's functionality required direct hardware access.
Eventually Arduino is going to need to provide a more complete API. It's so very unfortunate that the motivation to design the API with forward thinking towards future platforms hasn't been present all these years. So many very compelling sketches and libraries have been written over the last couple years. A great number that do anything "interesting" are unable to use only the Arduino functions without resorting to AVR-specific direct hardware manipulation.
Even today, lots of sketches work on Uno but not Mega. When/if Due is released, certainly simple LED blink examples will work, but it's going to be take a lot of work for many of the compelling libraries and sketches are ported.