Years ago, I developed a patch for Teensy to speed up compiling. It was contributed back to Arduino. I'm not sure if it's in 1.5.x, but it should be.
However, it may not work properly when directories with spaces or other special characters are used. The patch is designed to "fail safe", reverting to the old (slow) behavior of compiling every file. Only if several checks all pass does it use the previously compiled .o files. One of those checks involves parsing numerous files written by a previous run of the compiler, listing numerous full pathnames to files. If the pathnames have special characters, the compiler might add extra encoding, which the patch doesn't know how to handle. If the names don't match perfectly, it's designed to revert to the old behavior, which is slow but very safe. Since developing Teensy 3.0, investigating this and other non-critical bugs has fallen to the absolute bottom of my priority list. It'll probably be about a year until I investigate this. If anyone else wants to look at it sooner, the code is in is_alread_compiled() in Compiler.java.
In the meantime, knowing this, you might try placing the Arduino IDE in a directly with no spaces or special characters. A location like C:\Program Files (x86) has both spaces and parenthesis.
If you'd like to help on this issue, without digging into Compiler.java and arcane compiler options, it would be very helpful if someone were to meticulously investigate changing the directory name where Arduino is installed (trying many different non-alphanumeric characters) and check if the speedup does or doesn't work. Documenting what does and doesn't work, and on which platform (Linux, Mac and Windows may be different) would help others sooner, and give me or whoever eventually works on this a good start with solid info.
I can't help with Due's bootloader, but hopefully you can get my speedup patch working for a little relief from waiting for all those files to recompile every time?