Consider the Arduino chip itself. It has protection diodes on every input pin. Those diodes will conduct any time the pin voltage exceeds the power voltage. The excess voltage is directed to the power supply.
What happens when the power supply to that particular chip is off? The power supply pin is at zero volts. The diodes will conduct any time any input pin is over zero volts. That will prevent any other device on that bus from using the bus as it's dragged down to zero.
You have a mix of devices. Some/most of them have the same kind of protection on their inputs. You cannot power off the chip and leave it attached to the I2C bus.
Either change your whole plan or use a level-shifter circuit at each device which can be turned off. The level shifter can cope with the 'down' level being zero or equal to the 'up' level.