The typical scheme is 5V -> 10K resistor -> analog pin -> phototransistor -> GND (a "common emitter" scheme). This sets the pin high when it's dark and low when it's bright. I can't say I fully understand the math but it just seems to work better at typical light levels.With your current wiring a larger resistor should give you more of a sensor swing.
For starters you setup is totally wrong you want to feed the ptoto transistor with a 100k resistor VDD to collector and tap the ADC to the emittor with a 10k resistor
The way I posted was at the emitter side which should work fine the one youPosted is not going to have the range your looking for
I don't understand why Vishay didn't put any "typical configuration" schemes in datasheet. I googled a lot of pages about phototransistor but there's so many different schemes.