My guess is the slowness comes from several sources.
Obviously it's not even beginning data transfer for 2 seconds. Why, I have no idea.
My guess is those gaps are latency added mostly by the firmware in the 16u2 chip. I believe the upload protocol, which is created by Atmel and permanently burned into a non-upgradeable ROM inside the SAM3X chip, involves a command-response approach. That's never great, since there's always some latency (especially on Windows) from the USB frame times. But the 16u2 probably adds much more, because it probably waits for a timeout before sending any buffered serial data as a partial USB packet.
The protocol itself involves ascii encoding of data, and 115200 is only 11 kbytes/sec, so even without the dead times, the speed can't be great with serial.
Upload protocols and speeds are something of an obsession of mine. For Teensy 1.0 & 2.0, I used USB control transfers, but also still a 1-at-a-time approach requiring the control transfer's final ACK. That's reasonably fast, but still far from optimal. For Teensy 3.0, I tried a new approach that allows streaming with substantial buffering by the bootloader, and an ACK/NAK approach to allow the transmitter to sense the board's ability to accept more data. It turned out a lot of the limitation in speed was due to the operating system's latency in scheduling the userspace program to run. The streaming, buffering and ACK/NAK solves that problem and lets the upload happen at a speed paced only by the flash write timing (if you try a Teensy3, I think you'll be impressed how fast 100K uploads). Still, even with all those measures, detecting the request to upload, disconnecting from the USB and re-enumerating are slow, taking about 1 to 2 seconds. At least with Teensy they are, since the USB disconnects. I have a few ideas about speeding that up... but they're extremely difficult. Did I mention I tend to obsess about upload protocols and speeds?
Anyway, for Arduino Due, the upload speed could probably be improved considerably if someone put a lot of programming work into making the 16u2 chip more aware of the upload protocol. Actually, it probably has no need to be aware of the PC-to-Due stream... it's the Due-to-PC responses that it could recognize. If it could detect those and quickly transmit a partial USB packet, rather than sitting there waiting for more data, you'd probably see substantial boost in speed. It might also be possible to increase the baud rate? I don't know what baud rates Atmel's bootloader can support, but the 16u2 is theoretically capable of up to 2 Mbit/sec. In practice, 16 MHz AVR can do about 0.5 Mbit/sec with well written code in C while also polling the USB stuff.
But ultimately, some of the slowness is the non-optimal protocol Atmel designed, both the non-binary data format and the command-response nature. Unfortunately, there's nothing you can do about those issues, since the bootloader is permanently burned into a ROM on the chip that can never be upgraded.