You need to turn off the I2C on the unpowered devices. This can be done using back to back level shifters. See page 12 of this application note.
It looks complicated but you can use 2 common level shifter modules to build it.
Example is here.
EDIT: We used 4 channel converters because they were common and cheap. You only need 2 channels (one each for clock and data). There are 2 channel converters but they are less common. For example.