Some sensors are not that strong with pulling down the SDA and SCL, and those are the first that cause trouble with too much pullup.
I have a level shifter with pullups on both sides, plus sensor modules, plus the Arduion Mega board with onboard 10k pullups. It's just too much.
With a level shifter, a 3.3V sensor has to pull down the 3.3V side and the 5V side of the I2C bus. Also a 5V sensor has to pull both sides down.
I think about 2k2 total pullup can be used for a 400k I2C bus. But 4k7 for total pullup on a normal 100k I2C bus. The total value is all the pullup resistors parallel of course. With a level shifter in the circuit, the 3.3V makes it different, and it is better to calculate the total current.
So I de-solder all the pullups from the sensor modules, since I have enough pullup in my circuit already. I also disable the internal pullup resistors in my Arduino I2C slaves now.
P.S.: In Arduino 1.5.8 a new function Wire.setClockSpeed() can set the speed to 400k (or any other value).