Has anyone considered adding DMA support to the Due USB code? I'm really wondering why such a thing wouldn't have been done from the start. The SAM3X has support for direct transfer of data to and from USB right to RAM without the core processor having to be involved. That should drastically increase the maximum read/write speed while also freeing up processor time. It's a little more complicated but might be worth doing. Though, since I'm writing this post and not doing it myself I guess it isn't worth that much to me. 
But, if you're in the code anyway, give it a try. Once it's all spiffy and fast I'll try to help get it pushed into the mainline code.