I experienced once that there was a big time difference when dividing two int32 values compared to dividing two int64 values (don't remember the excact numbers, but was something like 1us to 40us). For all other operations it seemed to be like expected. Unless the division problem, in all my code examples int64 math was much quicker than float32 math and roughly two or three times slower than int32 math.
So the problem could be performing divisions using data types larger than the atomic size which is 32bit for the DUE. The division of float32's however should be somehow an atomic operation.
Any comments or tests on that?
*update: you can also find where exactly the time difference is caused in your code by using stuff like:
t1 = micros();
**code**
t2 = micros();
Serial.println(t2-t1);
for every block of code.. I guess you will find an int64 division to be the crucial point. could you try?