Go Down

Topic: How to safely and reasonably convert a float or double to string or char array? (Read 2058 times) previous topic - next topic

alirezasafdari

Hi

I have been using in dtosrtf() in a relatively big project and some usual stuff was happening. Long story short, I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory.

This is very surprising for me and for those who may have a hard time believing it, I have written a little code here.

Code: [Select]
volatile float numberInput = -3, zero = 0;
char buf[50];

void setup()
{
Serial.begin(115200);
Serial.println("begin");

numberInput = 99999999999999999999999999999999999.0;
dtostrf(numberInput, 5, 1, buf);
// expectation: it will not go beyond 5 character in the buffer
// reality: it goes and screw all the data in the memmory
// question: how do we do this safely?
Serial.write(buf, 50);
Serial.println();
}


And the result in serial port was shocking

Code: [Select]
100000000000000000000000000000000000.0..............
The . represent 0x00. (checked with non Arduino serial monitor). No surprise there.


And just to show how dangerous it can get, the following code it keeps printing "begi" in serial port. (it continuously start from the start of the program) (I just changed buffer size to 5 instead of 50)

Code: [Select]
volatile float numberInput = -3, zero = 0;
char buf[5];

void setup()
{
Serial.begin(115200);
Serial.println("begin");

numberInput = 99999999999999999999999999999999999.0;
dtostrf(numberInput, 5, 1, buf);
// expectation: it will not go beyond 5 character in the buffer
// reality: it goes and screw all the data in the memmory
// question: how do we do this safely?
Serial.write(buf, 50);
Serial.println();
}





When I was suspicious of dtostrf(), I taught looking at the implementation would help (I asked in forum). However, I was very wrong but in case that helps you to come up with a solution, you can check the implementation here:

http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/dtoa_prf.c?revision=1944&view=markup

By safe and reasonable, I am looking for a solution where I do not have to worry if I do not provide enough space. I prefer to get acknowledged that the operation was not successful rather than the whole memory being screwed. BTW I am not asking for a magical solution, so I let the function know how much space it has in the buffer, so that it does not go all over the place.

Regarding dtosrtf: I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.

Someone in another forum suggested using 100 byte array to be safe, but I think this is not a reasonable solution on a AVR/Arduino with limited memory.
CRAE TECH

pert

I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.
Width is a minimum, not a maximum. From http://www.nongnu.org/avr-libc/user-manual/group__avr__stdlib.html:
Quote
The minimum field width of the output string (including the possible '.' and the possible sign for negative values) is given in width
Try this:
Code: [Select]
void setup()
{
  Serial.begin(115200);
  Serial.println("begin");
  double numberInput = 9.0;
  char buf[50];
  dtostrf(numberInput, 5, 1, buf);
  Serial.print(buf);
}
void loop(){}

jremington

Code: [Select]
numberInput = 99999999999999999999999999999999999.0;You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.

With a bit of thought you can plan ahead when using dtostrf() and allocate a sensible amount of space. A 10 character array should suffice in reasonable situations. One character for sign and one for decimal point leaves 7 for significant figures - any more would be nonsense. Don't forget the terminating zero byte!

gfvalvo

Code: [Select]
numberInput = 99999999999999999999999999999999999.0;
You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.
It would be unreasonable even if a double really was twice as many bytes as a float.

alirezasafdari

I am sorry, with all due respect but this is how I look at it.

Is it unreasonable? yes.
Is it possible for the event to occur? yes.
How bad it is, if it happens? very bad.
I need a solution for this.

Specially consider the cases where the user is going to enter a number. Or even some part of the code, some how manage to produce a very large number, due to mistakes in code (debugging) or hardware error/faults (disaster).

Let's say a sensor connection come loose and the ADC reading drops to a level it would never happen if the sensor was attached to it. And for some reasons a constant is divided by this float variable coming out of sensor. Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved and then things may not look that good.

CRAE TECH

gfvalvo

Yes, the C / C++ language affords you many opportunities to shoot yourself in the foot if you're not paying attention - pointer manipulation being a prime example. User interaction and external hardware interfaces are always a wildcard. It's up to the programmer to determine how much error checking, limit testing, fail safe, etc to provide. Part of that decision is assessing how disastrous the results of a failure can be.

jremington

Quote
Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved
Evidently, you need to learn some things about input conditioning and error handling, which all of us working in the real world have to deal with on a day to day basis.

It is trivial to fix or work around the limitation of dtostrf.

1. Use snprintf() with floating point extensions enabled.

2. Work around the limitation of dtostrf() by thinking about the problem for 10 seconds.

For example:

Code: [Select]
void setup()
{
  Serial.begin(9600);
  Serial.println("begin");
  double numberInput = 99.0;
  fp_print(numberInput);
  numberInput=999999.0;
  fp_print(numberInput);
  numberInput=10.*numberInput;
  fp_print(numberInput);
}
void loop(){}

void fp_print(float x)  //function to print a float, within range
{
  char buf[10];
  if ( fabs(x)<9.99999E6 ) Serial.println(dtostrf(x, 5, 2, buf));
  else Serial.println("overflow");
}

Delta_G

Let's say a sensor connection come loose and the ADC reading drops to a level it would never happen if the sensor was attached to it. And for some reasons a constant is divided by this float variable coming out of sensor. Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved and then things may not look that good.
You are misunderstanding what a float or double variable can or cannot do.  Even if the number gets as large as 35 9's in a row, a float variable can't represent that exactly.  It's only going to be able to do 6 or maybe 7 digits of precision.  So you wouldn't want to print out all 35 digits.  They wouldn't be right anyway.  Just print out the 6 digits you know are good.  So represent it as 9.99999E34.  With dtostrf you get to specify how many digits to display.  So you'll know the maximum possible width of the string it produces and can plan your array accordingly.  Trying to capture all those digits is just a total waste of your time and effort. 
If at first you don't succeed, up - home - sudo - enter.

alirezasafdari

I am sorry if I am sounding arrogant but I really do not understand why you guys are trying so hard to say a problem does not exist when it does.

@jremington
For your solution 1, I do not know how the "extensions enabled" work and I could not find much information on sprintf in arduino except the fact that it does not support formatting. Will be more than happy to see a link or an elaboration of your idea.

For solution number 2, you gotta be kidding. Do you actually understand how much time it takes to check the float range for every conversion? Also, if you have ever rolled your eyes on dtostrf implementation, it knows exactly what is going to happen. it knows the exponent and can easily compute and estimate the length required right after its first step which is extracting the exponent. I have done this and considering the current attitude in the community so far, I guess I end up doing it myself and making a safe version of dtostrf (which could stop when buffer length is used up and return a false), or a conditional checking using the mathematical properties of exponent (less efficient because it will happen again in side dtostrf)


@Delta_G
I think you are focusing on the meaning of the number. Where I do not even consider that as an on topic . I understand your point, but if you have a heavy calculation in your program and some variables may happen to have an unusual values then having 35 digit number is possible. Yes, the data is in accurate and all that but it has happened and probably you do not know that it has happened in your program because you did not expected the unusual value when writing the code (yes, you can be careful and check stuff but there still might be cases where the number can get screwed up) and when the dtostrf is called it starts writing on the rest of the memory where it is not supposed to. That is not something you can forgive in most cases.
CRAE TECH

jimlee2

Quote
With dtostrf you get to specify how many digits to display.
I don't think that is true. If I remember correctly, you can only specify minimum size or digits. There is no maximum. I'm pretty sure I've run into what he's found and its not obvious how to shield yourself from this problem. 999--99.0 is one thing, but another is the 1.0000000009 kind of thing. That also will blow up your string buffer.

Granted I was coding a teensy when I ran into this, so maybe the Arduino itself handles this better?

-jim lee

Delta_G

No, apparently you don't understand how numbers work. There are no such thing as numbers that have more or less digits than other numbers. All numbers have an infinite number of digits. If you say 1 then I can say 1.0 and someone else can say 1.00.  Whether or not you write all the digits when you write the number back to the human doesn't matter.  The math isn't being done with the output of dtostrf.  The math is being done with the actual floating point number.  So you're not losing on your math.  Your math in your program is safe. The only thing that String is for is showing the human what the value is.  And if you try to show him 35 digits then you are lying to your user about the value.

If the result is some number with 35 digits why in the world would a user want you to output that as a 35 digit number with only 6 that matter and 29 that are basically just made up when you can give him scientific notation and only the part that's accurate and actually conveys the value. 

This problem isn't unique to computers.  Big numbers come up in science all the time and with propagation of error from measurements we know how many significant digits we actually have. And we don't ever write out 35 digit numbers there.  At least not unless we actually have 35 digits of precision. 

Sure, big numbers might come up and they come up all the time.  But that doesn't mean it makes sense to write them all the way out without scientific notation. 
If at first you don't succeed, up - home - sudo - enter.

BulldogLowell

Quote
All numbers have an infinite number of digits.
No, not the counting numbers (i.e. integers).

Delta_G

No, not the counting numbers (i.e. integers).
Yes, even them.  You have to separate the abstract concept of a number, from the concrete representation of the number.  For the abstract concept pretend I have 3 apples.  Would you write that I have 3, 3.0, 3.00, 3.000, 3.0000.   How many apples do I have there? 

Now the concrete is different.  If someone wrote that I had 3 apples, you aren't entirely sure I didn't have a little piece of an extra apple or that I hadn't taken a bite out of one of them and the writer rounded off.  But if he said that I had 3.0 apples, you could be sure that I hadn't bitten off more than a tenth of one.  If he said I had 3.00 apples then you're sure I haven't bitten off more than a hundredth of one. 

The key concept that the OP needs to grasp is the concept of significant digits.  Many a student has messed up their grade in chemistry class over sig figs. 

If at first you don't succeed, up - home - sudo - enter.

BulldogLowell

You are not correct regarding the counting numbers.

You are mixing oranges with your apple analogy.

Decimals do not exist in the set of integers, for example.

Delta_G

You are not correct regarding the counting numbers.

You are mixing oranges with your apple analogy.

Decimals do not exist in the set of integers, for example.
You're talking about representations of quantities. 

I am talking about numbers in the abstract.  Not the representation of quantity but the actual quantity itself.  I don't mean the part that you read and write, but the part that you actually do math with.   

If I have 3 apples and you have 3.0 apples, sure you represented yours with an integer, but we both have the same actual quantity of apples.  If we traded, it would still be an even trade.  Unless, of course, you were rounding. 
If at first you don't succeed, up - home - sudo - enter.

Go Up