Go Down

### Topic: decimal calculation correction problem (Read 4272 times)previous topic - next topic

#### Midway

##### Aug 11, 2013, 08:04 pm
hi,

this is the code:
Code: [Select]
` #define MF 38.9Na = Freq + MF;`

When i run the software in the Nano, I see that the decimal MF is 38.9000015258 and not 38.9.

how can i decrease this "15258" to 38.9

i have tried to:
Code: [Select]
`MFcorr = MF - 15258`

but not work well

i appreciate all the help
ed

#### Erdin

#1
##### Aug 11, 2013, 08:10 pm
Use this:
http://www.binaryconvert.com/result_float.html
Fill in "38.9" for the decimal, and click convert.
It is converted to a binary, but that binary is: 3.890000152587890625E1

It is a 32-bit floating point number, and that is the accuracy you get.

#### majenko

#2
##### Aug 11, 2013, 08:14 pm

hi,

this is the code:
Code: [Select]
` #define MF 38.9Na = Freq + MF;`

When i run the software in the Nano, I see that the decimal MF is 38.9000015258 and not 38.9.

how can i decrease this "15258" to 38.9

i have tried to:
Code: [Select]
`MFcorr = MF - 15258`

but not work well

i appreciate all the help
ed

Simple: don't use floats.  Use integer maths instead.

Instead of using the actual value, use the "milli" value.  For instance, instead of volts (3.2V) use millivolts (3200mV).  Instead of degrees C (25.6 DegC), use milliDegrees C (25600 mDegC).  Instead of Hz (143.2Hz) use milliHertz (143200mHz)...

#### GoForSmoke

#3
##### Aug 11, 2013, 10:09 pm

Simple: don't use floats.  Use integer maths instead.

Instead of using the actual value, use the "milli" value.  For instance, instead of volts (3.2V) use millivolts (3200mV).  Instead of degrees C (25.6 DegC), use milliDegrees C (25600 mDegC).  Instead of Hz (143.2Hz) use milliHertz (143200mHz)...

+1

Instead of measuring in decimal-point meters that would be measuring in millimeters.
If you need 6 places then use micrometers (millions of a meter).

int is good -32768 to 32767 which if all 9's gives you 4 places max (9999).
long is good -4294967296 to 4294967295 gives you 9 places of 9's (999999999).
long long (64 bit integer) gives you 19 places.

Put the decimal point in when you print the number out to keep your accuracy.
1002 mm = 1.002 m

1002 / 1000 = 1 .... print 1 then print .

1002 % 1000 = 2
2 / 100 = 0 .... print 0

2 % 100 = 2
2 / 10 = 0 .... print 0

2 / 1 = 2 .... print 2

That operation can be done in a while loop or a for loop.

2) http://gammon.com.au/serial <-- techniques howto
3) http://gammon.com.au/interrupts
Tasking: techniques make the trees, the sketch is the woods.
See the woods and the trees, comment both.

#### PeterH

#4
##### Aug 12, 2013, 01:49 am

When i run the software in the Nano, I see that the decimal MF is 38.9000015258 and not 38.9.

how can i decrease this "15258" to 38.9

I agree with the suggestions to use fixed-point arithmetic instead of floats. However, the original problem you described is not really about the value of the expression - it is only about how that value is represented as a string. There are various ways to control how floating point numbers, and which options are available to you will depend on how you're doing that representation - which you haven't shown us.

#5

#### GoForSmoke

#6
##### Aug 12, 2013, 05:19 am
I'll stick with Kernighan and Plauger: "Floating point numbers are like piles of sand; every time you move one you lose a little sand and pick up a little dirt."

http://introcs.cs.princeton.edu/java/91float/

Quote
If the errors occur at random, we might expect a cumulative error of sqrt(N) ?m. However, if we are not vigilant in designing our numerical algorithms, these errors can propagate in very unfavorable and non-intuitive ways, leading to cumulative errors of N ?m or worse.

When I worked on any code that involved money; billing, payroll, taxes, etc, I stayed away from floating point. When I coded for NC and CNC cutting machines, I stayed away from floating point.

Besides, Arduino has no FPU. Floating point crawls without an FPU. Even 64-bit integers is faster.

2) http://gammon.com.au/serial <-- techniques howto
3) http://gammon.com.au/interrupts
Tasking: techniques make the trees, the sketch is the woods.
See the woods and the trees, comment both.

#### Midway

#7
##### Aug 12, 2013, 09:22 pmLast Edit: Aug 12, 2013, 09:27 pm by Midway Reason: 1
Hi all,

this is the actual plan:

To tune a TV-tuner with I2C i need to calculate HEX "lsb" and "msb".

When i adjust the frequency in steps (MHz or KHz) the following code i have made to calculate the HEX adress MSB and LSB.

Xtal: DEC 16.0 is used with Xtal_a = 4.0 MHz or DEC 20 is used with Xtal_b 3.2 MHz.
MF: Mid Freqency DEC 38,9 MHz

Code: [Select]
` float Na;float Freq; int Nb; int Nc; int msb; int lsb;Na =Freq+MF;Nb =Na*Xtal;Nc =((INT)(Nb/256)) * 256;lsb =Nb-Nc;msb =(INT)(Nb/256);`

-------------------------
this is the actual reading when the software is running.

the selected frequency = 1 MHz (also with a strange number at the end "1192")

1.0000001192 = Freq
38.9000015258 = MF
16.0000000000 = Xtal
91 = Band
39.9000015258 = Na
638 = Nb
512 = Nc
7E = lsb DR2
2 = msb DR1

ed

#### KeithRB

#8
##### Aug 12, 2013, 09:55 pm
There is no correction. Just like when you convert 1/3 to decimal you get 0.3333333333..., there is no exact binary conversion from 38.9 base 10 to binary. You have to live with it or use integers, as has been pointed out. Welcome to computer science!

Do you have a datasheet for the tuner?

#9
##### Aug 12, 2013, 11:51 pm
When i adjust the frequency in steps (MHz or KHz) the following code i have made to calculate the HEX adress MSB and LSB.

That code also truncates the value.  The "strange numbers" at the end can be ignored because they never become part of the final value.

#### GoForSmoke

#10
##### Aug 13, 2013, 02:32 am
Quote

the selected frequency = 1 MHz (also with a strange number at the end "1192")

1.0000001192 = Freq

The strange number is .0000001192, which in your case is too small to affect the results.
Even if Freq equaled .9999998808 it wouldn't make much difference.

2) http://gammon.com.au/serial <-- techniques howto
3) http://gammon.com.au/interrupts
Tasking: techniques make the trees, the sketch is the woods.
See the woods and the trees, comment both.

#### Midway

#11
##### Aug 13, 2013, 09:23 pm
hi,

thanks for getting the Info.

When i go 20 Mhz higher this is the result:

20.0000400543 = Freq
38.9000015258 = MF
16.0000000000 = Xtal
91 = Band
58.9000396728 = Na
942 = Nb
768 = Nc
AE = lsb DR2
3 = msb DR1

The result when i go to 30 MHz:

30.0000782012 = Freq
38.9000015258 = MF
16.0000000000 = Xtal
91 = Band
68.9000778198 = Na
1102 = Nb
1024 = Nc
4E = lsb DR2
4 = msb DR1

this is only an example when it count from 1 MHz to 30 Mhz.

At this moment the actual frequency's (when selected) must be:
Code: [Select]
`#define LB 0xA1                   // Low frequency band 46.25 - 170 MHz = 0xA1 #define MB 0x91                   // Mid frequency band Mid 170 - 450 MHz = 0x91 #define HB 0x31                   // High frequency band High 450 - 860 MHz = 0x31 `

In my opinion it make much difference.

ed

#### Delta_G

#12
##### Aug 13, 2013, 09:40 pm
I don't understand the difference you think it makes.  When you make the conversion at the end, the extra part after the decimal is being truncated.  So you're getting the same answer in the end that you would have gotten with the exact numbers.
|| | ||| | || | ||  ~Woodstock

#### GoForSmoke

#13
##### Aug 14, 2013, 02:28 am
Midway, I wonder if you understand math or numbers?

20.0000400543 / 20 = 1.000002002715, the error is digits a float can't store but the print generates. A float (Arduino double and float are THE SAME) is only good to 6 or 7 digits which means you don't have 20.0000400543 but actually 20.00004. But SO WHAT?

30.0000782012 < 30.001 < 30.1 < 31.0

You have ranges like 46.25 to 170. 46.250099999 is less than 46.26 let alone 170!

The error is not significant. Where the digits are is more important than what the digits are.

If you need exact then use fixed-place integers.

And just BTW, I hope that you don't expect to measure 46.25 MHz and faster with an Arduino that runs at 16 MHz.