I have been doing some basic tests also. I just have the arduino echoing back each byte. It does seem to be glitchy, but it does run fast. I'd be interested to see what kind of tests you are running and the results.
I think that HID is the way to go in the future because of the driverless installation. But I think that wouldn't be possible using the FTDI chip to do USB-serial. I think it would have to be with the atmega-usb. As long as Arduino is still a USB-serial device, I don't think implemneting HID over serial would be worth the pain.