As some of you may recall, I have been working on a quadratic least squares approach to self-calibrate the temperature sensing system on a board I developed. The input is a 16-bit ADC, so the numbers it produces can be large. I wrote a excel spreadsheet to keep track of all the components that go into calculating the denominator and the various factors to ensure that they are never larger than 2^63.

I discovered that I had to right-shift the inputs into the quadratic process by 6 digits in order to stay below this limitation. One of the issues I discovered on the Arduino (and hence the Excel work) is that there is no way to print int64’s using the usual serial.print command. I tried to write a program to print them out a digit at a time (all 20) but the results are not exactly what I expected. (i.e. the results for a int64’s always starts with a 52000… for some strange reason.).

```
void print_UINT64(int64_t value)
{
uint8_t digits[21];
for (uint8_t i=21; i>0; i--)
{
digits[i]=value%10UL;
value=value/10UL;
}
for (uint8_t i=0; i<21; i++) Serial.print (digits[i]);
Serial.println();
}
```

In order to maximize the effectiveness of the calibration routine, I have to use resistors that span a wide range of resistances and the results (especially when using excel) are beautiful, i.e. calibration to within a LSB across the entire spectrum of resistances / ADC values. Even with the limits of an arduino (see below) allow me to get within 1 LSB at the low end of temperatures and within 4 LSBs at 40+*C for an error around 0.04*C.

I would like the unit to self-calibrate during use, but recognize that it might be better in terms of results, etc. to account for ADC drift by just registering the zero ohm offset and then using a previously-calculated offset function (with its 0.99999747 R^2 fit) to do the rest in open-loop. I just worry that using an open loop approach may not be that accurate if the ADC is in a hot or cold environment, etc. I’d have to test for that and see how it compares to the performance of the closed loop error-compensation on the Arduino.

In the snippet below, rounds refers to the number of rounds the for loop executes, i.e. the number of digits being evaluated in the least squares approach. There currently are 5 reference (0.1%) resistors that get queried and thus 5 rounds. The Get_ADC_value is a routine that addresses the ADC and returns a 16 bit unsigned number with the results (i.e. 0-65k). The Ideal_ADC_Values array stores the ‘ideal’ ADC values that the ADC should return for a given resistor. The routine then calculates the deviation thereof which then becomes the y component of the quadratic equation. The Calibration Channels array stores the MUX channels for the calibration resistors, so the Get_ADC_Value function knows what MUX channel to activate for a given expected resistance.

```
unsigned int x=0;
int y=0;
int64_t z=0LL,sum_x=0LL,sum_x2=0LL,sum_x3=0LL,sum_x4=0LL,sum_y=0LL,sum_xy=0LL,sum_x2y=0LL;
long a=0L,denominator=0LL;
//the following variable determines how many data points the calculation gets to consider
byte rounds = Calibration_Channels.count();
for (int i=0; i<rounds; i++)
{
x = ((Get_ADC_Value (Calibration_Channels[i]) >>ADC_Error_Right_Shift));
Serial.print(" x: ");
Serial.print(x);
y = ((Ideal_Calibration_ADC_Values[i] >>ADC_Error_Right_Shift));
y = y-x;
Serial.print(", y: ");
Serial.println(y);
sum_x+= x;
sum_x2 += x * x;
sum_x3 += x * x * x;
sum_x4 += x * x * x * x;
sum_y += y;
sum_xy += x * y;
sum_x2y += x * x * y;
}
```

x and y are calculating correctly, so I am very happy with the two results. The only improvement opportunity I can think of the top of my head is adding 2^(right_shift-1) to each result before right-shifting in order to assure that the routines rounds up and down correctly, right?

The quadratic equation components then are fed into the following set of formulas which uses a int64 for z. Since the numbers being returned by the quadratic equations are very large, I right shift all of them 34 places and store the results in a long (a). I then attempt to divide the two longs to get the three outputs / factors from the quadratic equation (i.e. the multiplier for x^2, x, and the offset). Here, I fail miserably, as the results are not floats as I expect them to be.

```
z = rounds * (sum_x2 * sum_x4 - sum_x3 * sum_x3) - sum_x * (sum_x* sum_x4 - sum_x2 * sum_x3) + sum_x2 * (sum_x * sum_x3 - sum_x2 * sum_x2);
denominator = -(z>>34);
Serial.print("denominator:");
Serial.print(denominator);
z = (sum_y * (sum_x2 * sum_x4 - sum_x3 * sum_x3) - sum_xy * (sum_x * sum_x4 - sum_x2 * sum_x3) + sum_x2y * (sum_x * sum_x3 - sum_x2 * sum_x2));
a = -(z>>34);
Serial.print(", raw offset:");
Serial.print(a);
ADC_Corrector_Offset = (float)(a/denominator);
z = (rounds * (sum_xy * sum_x4 - sum_x3 * sum_x2y) - sum_x * (sum_y * sum_x4 - sum_x2y * sum_x2) + sum_x2 * (sum_y * sum_x3 - sum_xy * sum_x2));
a = -(z>>34);
Serial.print(", raw x:");
Serial.print(a);
ADC_Corrector_x = (float)(a/denominator);
z = (rounds * (sum_x2 * sum_x2y - sum_xy * sum_x3) - sum_x * (sum_x * sum_x2y - sum_y * sum_x3) + sum_x2 * (sum_x * sum_xy - sum_y * sum_x2));
a = -(z>>34);
Serial.print(", raw x^2:");
Serial.print(a);
Serial.println();
ADC_Corrector_x2 = (float)(a/denominator);
```

So, any pointers as to what I am doing wrong regarding the divisions to not make the results a float? Also, what is a viable way to print out a INT64? Many thanks everyone in advance. Constantin