This is very cool-- been wondering if you could make a fully-self-contained RNG without sensors. Still worried about it though. I wouldn't depend on it as anything more than an incremental amount of entropy in a larger system.
So I wrote a program that calibrates itself. Thus if the signal changes, the micro-controller adapts. It fixed the problem.
Recalibrating in software, without any adjustment to the circuit, may postpone the problem but I don't think it can fix it. Just my thinking, but if the median 1/0 ratio continues to shift, you're using fewer and fewer of the bits of the ADC, and eventually you'll be left with a very sloppy all-zero (or all-one) signal.
What are the attacks on this RNG? Obviously we can assume it doesn't work well if it's physically damaged, but what about undetectable temporary stresses like a blast of freon or a hairdryer over the board?