The sensor output voltage only changes by 10mV per degree, so its vital that you measure the voltage across the sensor, not the voltage across the sensor plus the voltage along a wire carrying current. This means the ground wire to the sensor must run separately from any other ground wiring. You are probably just seeing the IR losses along the ground wiring to the LED.
The Arduino has several ground connections available, reserve one for analog sensors only, use different ones for anything else.
However having said that you are seeing about 0.11V change, which is more than I'd expect - perhaps your 5V rail is not regulated? The ADC only measures relative to its reference voltage, which is (by default) the 5V rail. If you power the Arduino from the barrel jack then the internal 5V regulator will be operational and should ensure clean power rail voltage.