Arduino Uno with two i2c color sensors

Hi, I'm having trouble with an Arduino Uno that has two i2c color sensors connected to it. These are the sensors: Color Light Sensor Evaluation Board - SEN-10701 - SparkFun Electronics
In the hookup the grounds are connected together from both sensors and the SDA and SCL are switched between either of the color sensors by a 74HCT4053N (the color sensors have exactly the same adress and it can't be changed).
The problem is that when either of the sensors are connected with its SDA or SCL straight to the Arduino (so without the 74HCT4053N switch) it works just fine and reads color data, while with the switch suddenly no readings appear in the serial monitor anymore, while only one SDA and SCL are allowed to pass through and it should effectively give the same result as connecting these wires straight to the Arduino. What is wrong here?

Do you have the schematic? How are the pull-up resistors wired? Is the Enable pin pulled low?