For Arduino Uno & Mega (and clones), you are correct. The slow rate of serial data limits how fast things can run.
For Arduino Due (native port) and Teensy 3.1, and even the slower Arduino Leonardo and Arduino Zero (native port) and Teensy LC & 2.0, the communication runs at the full USB speed. It is NOT limited by a serial baud rate. On those boards, regardless of the number used with Serial.begin() or SerialUSB.begin(), the communication always occurs as fast as the USB ports on both ends can deliver data.
If you've only ever used Arduino Uno & Nano (or clones), you've probably never experienced the speed of modern USB-based boards. They've very fast.
Before Arduino 1.6.0, the Arduino Serial Monitor would consume all the Java virtual machine memory in a matter of minutes. Massive CPU usage occurred too, effectively crashing the Arduino IDE on many slower computers, before the memory crash could happen.
I personally redesigned the serial monitor, shortly before Arduino 1.6.0. Others had tried. Many attempts to limit memory usage caused a lot more CPU usage, which made the crashes even worse. A lot of difficult work went into making the Java-based serial monitor able to keep up with maximum rate data from these faster boards. Especially Teensy 3.1 could massively overwhelm the older Arduino IDEs (even though Due should be theoretically faster).
If you've never used any of these faster boards, I highly recommend you get one and try testing your code with the fast rate of data it can send. Slow serial-based boards are still widely used, but already fast native USB boards are less expensive (eg, Teensy LC or Leonardo clones) than a serial board with a USB-serial converter, so native USB is certainly the future.
If you're developing and testing new software, don't you think it'd be a good idea to test with modern native USB hardware, rather than only older and slow baud rate limited serial?