Why is DUE port access so slow?

This won't happen while using INPUT_PULLUP (approx 50K-100K) ... need lower value (i.e 2.2K) external pullup resistors.

Do you mean, I've to build my own pullups (2.2K) and not to use software pullups?