The millis() timer on the Arduino isn't 100% accurate. it fluctuates +/- 1 ms (?) at any time, but maintains accuracy over time.
So any millis time measurment could be up to 1 mSec out and rarely spot on to the microsecond.
If you can do your comparisons using elapsed microseconds(), which is 100% accurate all of the time. (subject to the crustal or resonator accuracy)
The millis() accuracy is related to the way it is implemented. A search will provide the details.
This probably expalins what you observed!