Which Arduino type are you using? The exact pin assignment for I²C connections varies depending on the board. The Uno has them on A4 and A5 as far as I know, while the Mega uses D20 and D21 (which are also explicitly labelled as SDA and SCL).
Did you connect the sensor directly or using the Sparkfun breakout board? When I tried to use it, I got the impression that connecting the Sleep and ExtClk pins (even without actually using them) causes malfunctions in the sensor (it got very
hot very quickly).
You might also want to have a look at my code (https://adjdcolourarduino.codeplex.com/
). After unsuccessfully trying to adapt one of the examples I found online to my existing code, I decided to start over, and included quite a few (mostly uncommented, sorry) functions that might make the ADJD-S311 easier to use.