I have gotten the sensor to work without level shifting it (as sda/scl are open collector).
Yes they are but what are they being pulled up to?
If it is more than 3V3 at the sensor end then they need some level translation. If there is no pull ups then you need some, but only to 3V3 on the Due end.