Ok, well, this is strange. I changed all of the doubles to floats. The Mega had exactly the same time (which reinforces your point). However the time for the Due actually went UP.
For due...
double: 1785 ms
float: 2056 ms
I haven't had time to go through the floating point library, but do you suppose the Due handles all floating points as doubles by default?
Steve.