Unfortunately, double is simply a synonym for float on the Arduino, and a quick look online showed no library for higher precision floating point (but it was a quick search - if anyone finds one, I too would like to know).
The float is a 32 bit variable, which ends up being about 6-7 digits of precison, and would explain the problem here:http://www.arduino.cc/en/Reference/Float
For example, even your value of inc (0.1) won't store precisely, so you'll have problems.
One thought - convert these to longs:
long c0offset = -60;
long c1offset = 0;
long c2offset = 60;
long d01 = c1offset - c0offset;
long d02 = c2offset - c0offset;
Since they all seem to be integer anyways, and then change the formula:
x01 = ((sq(d01) - sq(Rcircle1) + sq(Rcircle0)) / (2.0 * d01)) + c0offset;
x01 = ((temp2 - sq(Rcircle1) + sq(Rcircle0)) / temp ) + c0offset;
It won't do much, but you'll save two floating point multiplies, and resultant loss of precision.