Teensy3 may or may not work for your display needs. It does have a very optimized USB stack. For full disclosure, I'm the author.... so my opinion may be biased.
On the Ethernet low-level speeds, the W5100 always has 75% overhead. Every 1 byte access has 3 overhead bytes. So in a best case scenario, the fastest it can achieve is 25% of the SPI speed.
The exception is UDP, where block read is used to good effect. The one alternative that might be very fast with Arduino Due is a W5200 using UDP. Currently the only W5200 support is a simple modification to the existing library published by the Wiznet folks (who make the chip). It does use the efficient block transfer, but does nothing to optimize higher levels in the library. Since UDP is already pretty efficient, that might be reasonably fast, especially if you do some hacks to increase the SPI clock speed.
So what you say is that UDP with the w5200 is easy to integrate? At least without further optimizations?
Well, no, not exactly. I'm saying it might be faster, when used with Due and the existing (not very well optimized) Ethernet files for W5200. Maybe?I haven't personally tested this. But if you get a W5200 (eg, a Wiz820io module) and connect it to an Arduino Due and replace w5100.cpp and w5100.h with the W5200 versions (as published by Wiznet), I'd be curious to hear how fast it actually is?