Rounding Strangeness

G'day all,

I've observed unusual rounding behaviour when grabbing the co-ords from an SKM53 Skymaster GPS unit.

The data is returned (using TinyGPS) as a whole number, ie -3792166

The example uses serial.print to display the data, in thousandths of a degree, rounded out to 7 decimal places :

Serial.print(pLAT/100000.0,7);

This returns -37.9216613

If I divide -3792166 by 100000, to 7 places, I'd expect to see -37.9216600

Can anybody tell me where the 13 on the end of the return result comes from?

The additional numbers are required for accuracy, I just can't work out how to produce them outside of Serial.print.

Any and all assistance is greatly appreciated.

Cheers,

Melbfella.

A float has around 7 digits of precision, which is what you are getting. Anything after that is an artifact.

Hi Nick,

Thanks for your reply.

I'm not sure how your answer is relevant to my scenario - my maths is pretty rusty, but, regardless of precision, the '13' that is being banged into the result when using Serial.print to display it, simply does not exist in the original number.

Sorry for the stupid q's, but this is doing my head in. In the end, all I want to do is produce the same result, including the '13', without using Serial.print so I can log it to an SD card.

Cheers,

Melbfella.

Because float 32bit is with 7digits precision, and you printed 2digits+7decimal places = 9digits, therefore the last 2 digits are the artifacts.
So you have to "Serial.print(pLAT/100000.0, x);"
where x = (7 - number_of_digits_of_the_integer_part)

If you want it to be faithful to the integer value, without artifacts, report it as nn.ddddd00, installing the decimal place yourself. Of course, that isn't 7 decimal places of precision, but then neither is the result from the divide, with its artifacts.
You pretty much have to deal with the numbers you get, and all you have is seven digits to work with. Extending it to 9 digits is just pretending you have more precision than you have.

It's either that, or you could rewrite the Arduino librariy's float math functions.

It's an artifact of the printing. For example:

  Serial.print(pLAT/100000.0,15);

Gives:

-37.921661376953125

Those extra digits aren't "really there". It is just the way converting floats to strings work.

If you want precision you could use big numbers:

There is a new lib from Rob which you may consider:
http://forum.arduino.cc/index.php?topic=179111.msg1373140#msg1373140
It formats printf with #digits.

Simple solution:

  Serial.print(pLAT/100000.0, 5);

Prints:

-37.92166

Problem solved.

@Nick: see my above hint :slight_smile:
x = 7 - 2 = 5

Good point, and that will teach me to skim-read. :wink:

If I were you, I would double-check that that conversion is correct.

My GPS devices return the latitude as a number which represents degrees and then minutes and fractions of minutes.

So 37300000 would actually be 37.5 degrees