Smajdalf:
Using 1023 divider you read 0V from 0V to 5/1024V. You read 5/1023V for input from 5/1024V to 10/1024V etc. But you are right the returned value is "drifting" in the correct range.
Not following you here. Where did you get 10 volts from?
I'm only looking at the math.
count 0 = 0/10235 = 0
count 1 = 1/10235 = 4.887586mV
count 1023 = 1023/1023 * 5 = 5V
As you can see count=0 returns the low end of the first range but count=1023 returns the top end of the last range. From that it can be concluded the values in the middle are on a sliding scale. Such that at count=512 the voltage computes to 512/10235 = 2.5024437V; just a smidgen higher than 512/10245=2.5V but not quite to the upper bound value.
Smajdalf:
The datasheet does not say you should use 1024 and so you cannot get 5V! It just says 1023 represents voltage higher than Vref-1LSB. It is up to you what value you assign to this range.
The value assigned to the voltage range where count = 1023 is Vref-1LSB.
The conversion formula in the datasheet is the count divided by the count range times Vref.
count 0 = 0/10245 = 0
count 1 = 1/10245 = 4.8828124mV
count 1023 = 1023/1024 * 5 = 4.995117V (which is exactly what the datasheet says)
Unless you put an if statement to check for 1023 and force it to return 5.