How do you prevent ICs that are "turned off" from drawing parasitic power?

Put a P-channel MOSFET between 5V and I2C pullup resistors. Maybe have it between everything you want to power down even.