Help with int_64's...

As some of you may recall, I have been working on a quadratic least squares approach to self-calibrate the temperature sensing system on a board I developed. The input is a 16-bit ADC, so the numbers it produces can be large. I wrote a excel spreadsheet to keep track of all the components that go into calculating the denominator and the various factors to ensure that they are never larger than 2^63.

I discovered that I had to right-shift the inputs into the quadratic process by 6 digits in order to stay below this limitation. One of the issues I discovered on the Arduino (and hence the Excel work) is that there is no way to print int64’s using the usual serial.print command. I tried to write a program to print them out a digit at a time (all 20) but the results are not exactly what I expected. (i.e. the results for a int64’s always starts with a 52000… for some strange reason.).

void print_UINT64(int64_t value)

uint8_t digits[21];
for (uint8_t i=21; i>0; i--)

for (uint8_t i=0; i<21; i++) Serial.print (digits[i]);


In order to maximize the effectiveness of the calibration routine, I have to use resistors that span a wide range of resistances and the results (especially when using excel) are beautiful, i.e. calibration to within a LSB across the entire spectrum of resistances / ADC values. Even with the limits of an arduino (see below) allow me to get within 1 LSB at the low end of temperatures and within 4 LSBs at 40+C for an error around 0.04C.

I would like the unit to self-calibrate during use, but recognize that it might be better in terms of results, etc. to account for ADC drift by just registering the zero ohm offset and then using a previously-calculated offset function (with its 0.99999747 R^2 fit) to do the rest in open-loop. I just worry that using an open loop approach may not be that accurate if the ADC is in a hot or cold environment, etc. I’d have to test for that and see how it compares to the performance of the closed loop error-compensation on the Arduino.

In the snippet below, rounds refers to the number of rounds the for loop executes, i.e. the number of digits being evaluated in the least squares approach. There currently are 5 reference (0.1%) resistors that get queried and thus 5 rounds. The Get_ADC_value is a routine that addresses the ADC and returns a 16 bit unsigned number with the results (i.e. 0-65k). The Ideal_ADC_Values array stores the ‘ideal’ ADC values that the ADC should return for a given resistor. The routine then calculates the deviation thereof which then becomes the y component of the quadratic equation. The Calibration Channels array stores the MUX channels for the calibration resistors, so the Get_ADC_Value function knows what MUX channel to activate for a given expected resistance.

  unsigned int x=0;
  int y=0; 
  int64_t z=0LL,sum_x=0LL,sum_x2=0LL,sum_x3=0LL,sum_x4=0LL,sum_y=0LL,sum_xy=0LL,sum_x2y=0LL;
  long a=0L,denominator=0LL;
  //the following variable determines how many data points the calculation gets to consider
  byte rounds = Calibration_Channels.count();

for (int i=0; i<rounds; i++)
    x = ((Get_ADC_Value (Calibration_Channels[i]) >>ADC_Error_Right_Shift));
    Serial.print(" x: ");
    y = ((Ideal_Calibration_ADC_Values[i] >>ADC_Error_Right_Shift));
    y = y-x; 
    Serial.print(", y: ");
    sum_x+= x;
    sum_x2 += x * x;
    sum_x3 += x * x * x;
    sum_x4 += x * x * x * x;
    sum_y += y;
    sum_xy += x * y;
    sum_x2y += x * x * y;

x and y are calculating correctly, so I am very happy with the two results. The only improvement opportunity I can think of the top of my head is adding 2^(right_shift-1) to each result before right-shifting in order to assure that the routines rounds up and down correctly, right?

The quadratic equation components then are fed into the following set of formulas which uses a int64 for z. Since the numbers being returned by the quadratic equations are very large, I right shift all of them 34 places and store the results in a long (a). I then attempt to divide the two longs to get the three outputs / factors from the quadratic equation (i.e. the multiplier for x^2, x, and the offset). Here, I fail miserably, as the results are not floats as I expect them to be.

  z =  rounds * (sum_x2 * sum_x4 - sum_x3 * sum_x3) - sum_x * (sum_x* sum_x4 - sum_x2 * sum_x3) + sum_x2 * (sum_x * sum_x3 - sum_x2 * sum_x2);
  denominator = -(z>>34);
  z = (sum_y * (sum_x2 * sum_x4 - sum_x3 * sum_x3) - sum_xy * (sum_x * sum_x4 - sum_x2 * sum_x3) + sum_x2y * (sum_x * sum_x3 - sum_x2 * sum_x2));
  a = -(z>>34); 
  Serial.print(", raw offset:");
  ADC_Corrector_Offset = (float)(a/denominator);
  z = (rounds  * (sum_xy * sum_x4 - sum_x3 * sum_x2y) - sum_x * (sum_y * sum_x4 - sum_x2y * sum_x2) + sum_x2 * (sum_y * sum_x3 - sum_xy * sum_x2));
  a = -(z>>34); 
  Serial.print(", raw x:");
  ADC_Corrector_x = (float)(a/denominator);
  z = (rounds  * (sum_x2 * sum_x2y - sum_xy * sum_x3) - sum_x * (sum_x * sum_x2y - sum_y * sum_x3) + sum_x2 * (sum_x * sum_xy - sum_y * sum_x2));
  a = -(z>>34); 
  Serial.print(", raw x^2:");
  ADC_Corrector_x2 = (float)(a/denominator);

So, any pointers as to what I am doing wrong regarding the divisions to not make the results a float? Also, what is a viable way to print out a INT64? Many thanks everyone in advance. Constantin

you might need to add the DECimal flag?

for (uint8_t i=0; i<21; i++) Serial.print (digits[i], DEC);

I made a small sketch with your function and found a couple of problems in this statement.

for (uint8_t i=21; i>0; i--)

The first is that it should be i>=0 because otherwise it won't generate digits[0] - which is probably why you were getting 54 on the front. Secondly, I had to change "uint8_t i=21" to "int i=21" to get it to compile properly. With uint8_t the program compiled to about a 1.7kB program which didn't work. With "int" it compiled to around 11kB and did work. So, change that statement to:

for (int i=21; i>=0; i--)

and try again.


Thank you both.

The limitations of the 64 bit numbers being what they are, I have decided to do post processing (if necessary) in excel.

For now, the reference resistors will simply indicate if the corrected ADC results are in range or not. If the ADC results go wonky, that condition is noted on the SD Card file and post-processing has to happen within Excel. Certainly not as elegant as a self-calibrating ADC setup but likely to work just fine as long as the DAQs are not put into environments that are very hot or very cold.

Know of any 16-bit or 32-bit microprocessors that is "Arduino IDE compatible" and available in a TQFP-32 package? The only option I could think of is a PIC using the IDE that microchip is developing.