Can you show us the op-amp circuit you tried?
Gain of 50 to get .1V up to 5V should not be hard to do.
Another option is to use higher bit resolution external A/D.
If you had the internal 1.1V reference, than 1 bit represents 1.07mV,
so you should be able to get readings of 0 to 100 (0x64) with 100mV signal.
I dont follow your 10mV resolution comment.