Mixing 3V3 and 5V devices on the same 5V I2C bus is not going to work. If you see it functioning then you are killing the lower voltage device.
What you need is an analogue switch to swap over between the two busses with your pull up on the device side and an I2C libary that does not enable the internal pull up resistors.