And SCL/SDA directly to Pins 20 and 21 ? I measured the 2 pins from the interface, while powered from 5V and they have 5V each to GND.
You should level shift the voltage levels from the 5V gadget to the DUE's 3V level.
There are chips specifically designed to do this for you. It is the Right Thing To Do.
There is a possibility of your 5V gadget working fine with the I2C lines operating at 3V. If you can remove any pull-up resistors on the gadget's I2C lines (I2C lines are supposed to be open-collector), and use only the existing pull-ups on the DUE (which are tied to 3V), then that may be a working, however marginal, solution. You'll need to read the datasheets very carefully to see if the voltage thresholds are compatible. There have been numerous discussions on this already on this forum (do some searching).
If your gadget does not have pull-up resistors (and its I2C lines are configured as push-pull), then your gadget has an improperly designed I2C interface, and its designers should be flogged.
Running the DUE with the I2C lines pulled to 5V may work - for a while. However, this condition is out-of-spec, and may destroy the IO on the SAM chip. Then all you'll have is a dead DUE.