Its going to cause issues, yes.
Try putting 4k7 pull ups from SCL and SDA to +3V3 only. This might get things working. But it relies on all the 5V devices having a low enough VIH to recognise 3.3V as HIGH. If not you could also add extra 15k pull ups to +5V thus pulling the lines up to about 3.7V (which should be just OK for the 3.3V device).