Thanks Julian.
Two things:
I believe you are using the "internal" "2.56V" reference (Vref). Thats not a good idea, as the values you are measuring are potentiometric, not fixed. (case 2 on this page)
also if your "unknown" exceeds 10 ohms - or worst case goes open circuit - Vin will then be greater than Vref, and you wont get a true reading.
I'm sure using "DEFAULT" will give you better results.
FYI some "beginner mistakes" in a production unit:
using dupont wires
resistors too close together will not be able to dissipate their design power rating
wires floating about and not screened.
thin wires carrying current.
crimp push-on connectors not supporting the cable (the sleeve should also be crimped)
Ideally you would use a pcb.
As a rule, if it LOOKS untidy (like the above) it wont work well.
Good luck with your redesign, if you need any help just let me know. (click on my pic and message me)