Just adding some info at this point as I had to trawl through the whole thread to find it;
the actual has a infrared LED and a fototransistor.
Gustav, you need to get the sensor part working first. You also need to understand the behaviour of the devices you are using.
When un-illuminated a good (ideal) phototransistor should have infinite resistance -as no carriers are being generated at the junction.
Hence there is NO POINT in making measurements with it unlit. Its no surprise your readings are high.
Really it would be better to
measure the current through the phototransistor, so connect it to +5 and your resistor to ground.
Try a 10k resistor.
Then point the LED at the phototransistor.
Check how the value changes with LED on & LED off.
Come back to us with ..
1: DETAIL what are the device part numbers?
2: REVISED SCHEMATIC - so we can see what you have done
3: ANSWERS - what results did you get?
Meanwhile read this