 # Weird float results from calculations with longs?

I’m trying to use linear regression to calibrate a touchscreen but am getting some weird results. I was having an overflow issues when the ‘regPoints’ array was set to int, so changed it to long. I know I should have casted rather than change the data type, but anyway, it cleared up the overflow issue.

Now I think I’m having similar issues with my floats.

Here’s the test code with the regpoints hard coded so I can test the maths:

``````void calcTest(){
long temp;
long regPoints;
regPoints = 138;
regPoints = 283;
regPoints = 137;
regPoints = 650;
regPoints = 726;
regPoints = 610;
regPoints = 735;
regPoints = 282;
regPoints = 435;
regPoints = 463;

//  for (byte i = 0; i < 5; i++) {
//    Serial.print(regPoints[i],DEC);
//    Serial.print(", ");
//    Serial.println(regPoints[i],DEC);
//  }
double sigmaX = ((regPoints + regPoints)/2) + ((regPoints + regPoints)/2) + regPoints;
Serial.print("sigmaX ");
Serial.println(sigmaX,DEC);
int sigmaY = 15 + 111 + 63;
Serial.print("sigmaY ");
Serial.println(sigmaY,DEC);
long sigmaXY = (15 * (regPoints + regPoints)/2) + (111 * (regPoints + regPoints)/2) + (63 * regPoints);
Serial.print("sigmaXY ");
Serial.println(sigmaXY,DEC);
long sigmaXX = pow(((regPoints + regPoints)/2),2) + pow(((regPoints + regPoints)/2),2) + pow(regPoints,2);
Serial.print("sigmaXX ");
Serial.println(sigmaXX,DEC);
float touchXGrad = ((3 * sigmaXY) - (sigmaX * sigmaY)) / (3 * sigmaXX - pow(sigmaX,2));
float touchXInt = (sigmaY - touchXGrad * sigmaX) / 3;
Serial.print("xInt ");
Serial.println(touchXInt,DEC);
}
``````

And here’s the Serial output:

``````sigmaX 1302
sigmaY 189
sigmaXY 110552
sigmaXX 740894
xInt -7
``````

So, leaving the rounding issues in the calculations to one side, my final float results ‘xGrad’ and ‘xInt’ are being truncated.
I’m expecting results in the regions of 0.161888 and -7.31339 respectively.

How can I sort this out?

Cheers,
The Cageybee

``````Serial.println(touchXInt,DEC)
``````

Ten decimal places from a "float" is a bit optimistic, isn't it?

I was having an overflow issues when the 'regPoints' array was set to int, so changed it to long. I know I should have casted rather than change the data type, but anyway, it cleared up the overflow issue.

Since the regPoints values are well within the limits of an int, changing it's type was not necessary.

Casting might be, but I doubt it. Using intermediate variables of the right type is much more likely to solve the problem.

``````  double sigmaX = ((regPoints + regPoints)/2) + ((regPoints + regPoints)/2) + regPoints;
``````

Dividing by 2.0 would be much more useful, since this forces floating point arithmetic to be used, rather then integer arithmetic.

``````  int sigmaY = 15 + 111 + 63;
``````

Why is sigmaX a double and sigmaY an int?

Using pow to square a number is not efficient. The pow function expects both arguments to be floats, and is not smart enough to recognize that X * X is being performed.

``````  float touchXInt = (sigmaY - touchXGrad * sigmaX) / 3;
``````

Again with the integer arithmetic and expecting float results. Use 3.0!

Are you SURE the sketch uploaded properly after you made the changes? If the values you were outputting were floats (or doubles) there would be spme decimal places displayed, even if they were all zero. Since you are passing the value DEC (defined as 10) you should get ten digits after the decimal point in the output, even if they were all zeros.

Thanks for all the suggestions. I'll have a play and see what I come up with.

Why is sigmaX a double and sigmaY an int?

sigma Y will always be integers as they're the pixel references where I'm drawing the calibration points on the LCD. No need for a calculation there really.

sigma x should really be a float I guess as there's no difference between floats and duobles in Arduino. Also, as you point out, I should be using 2.0 or 3.0, a decimal anyway, which should sort out my rounding issues.

Again, many thanks. I'll post back with the results!

@johnwasser I'm actually not uploading the sketch as such, but running it in Oshon AVRSimulator.

The Cageybee

Well, it’s sorted. Many thanks for pointing out my school boy errors! So the code:

``````void calcTest(){
long regPoints;
regPoints = 138;
regPoints = 283;
regPoints = 137;
regPoints = 650;
regPoints = 726;
regPoints = 610;
regPoints = 735;
regPoints = 282;
regPoints = 435;
regPoints = 463;

float sigmaX = ((regPoints + regPoints)/2.0) + ((regPoints + regPoints)/2.0) + regPoints;
Serial.print("sigmaX ");
Serial.println(sigmaX,DEC);
int sigmaY = 15 + 111 + 63;
Serial.print("sigmaY ");
Serial.println(sigmaY,DEC);
long sigmaXY = (15 * (regPoints + regPoints)/2.0) + (111 * (regPoints + regPoints)/2.0) + (63 * regPoints);
Serial.print("sigmaXY ");
Serial.println(sigmaXY,DEC);
long sigmaXX = pow(((regPoints + regPoints)/2.0),2) + pow(((regPoints + regPoints)/2.0),2) + pow(regPoints,2);
Serial.print("sigmaXX ");
Serial.println(sigmaXX,DEC);
float touchXGrad = ((3.0 * sigmaXY) - (sigmaX * sigmaY)) / (3.0 * sigmaXX - pow(sigmaX,2));
float touchXInt = (sigmaY - touchXGrad * sigmaX) / 3.0;
Serial.print("xInt ");
Serial.println(touchXInt);
}
``````

And the results:

``````sigmaX 1303
sigmaY 189
sigmaXY 110553
sigmaXX 741761
Again, many thanks guys. You’ve saved me many hours of hair pulling!!! 