By handling serial in a non-arduino style, do we lose any critical feature of the arduino?
If you use the approach in wesfw's link you lose the Arduino Serial runtime support
My guess is that implementing a vga driver to the OP's requirements would eliminate most of the Arduino functionality. You could use the arduino IDE to create and download the code, but I would think that the running code would not have access to any of the Arduino specific input/output or time functions.