How to safely and reasonably convert a float or double to string or char array?

Hi

I have been using in dtosrtf() in a relatively big project and some usual stuff was happening. Long story short, I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory.

This is very surprising for me and for those who may have a hard time believing it, I have written a little code here.

volatile float numberInput = -3, zero = 0;
char buf[50]; 

void setup()
{
	Serial.begin(115200);
	Serial.println("begin");

	numberInput = 99999999999999999999999999999999999.0;
	dtostrf(numberInput, 5, 1, buf); 
	// expectation: it will not go beyond 5 character in the buffer
	// reality: it goes and screw all the data in the memmory 
	// question: how do we do this safely?
	Serial.write(buf, 50);
	Serial.println();
}

And the result in serial port was shocking

100000000000000000000000000000000000.0..............
The . represent 0x00. (checked with non Arduino serial monitor). No surprise there.

And just to show how dangerous it can get, the following code it keeps printing "begi" in serial port. (it continuously start from the start of the program) (I just changed buffer size to 5 instead of 50)

volatile float numberInput = -3, zero = 0;
char buf[5]; 

void setup()
{
	Serial.begin(115200);
	Serial.println("begin");

	numberInput = 99999999999999999999999999999999999.0;
	dtostrf(numberInput, 5, 1, buf); 
	// expectation: it will not go beyond 5 character in the buffer
	// reality: it goes and screw all the data in the memmory 
	// question: how do we do this safely?
	Serial.write(buf, 50);
	Serial.println();
}

When I was suspicious of dtostrf(), I taught looking at the implementation would help (I asked in forum). However, I was very wrong but in case that helps you to come up with a solution, you can check the implementation here:

http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/dtoa_prf.c?revision=1944&view=markup

By safe and reasonable, I am looking for a solution where I do not have to worry if I do not provide enough space. I prefer to get acknowledged that the operation was not successful rather than the whole memory being screwed. BTW I am not asking for a magical solution, so I let the function know how much space it has in the buffer, so that it does not go all over the place.

Regarding dtosrtf: I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.

Someone in another forum suggested using 100 byte array to be safe, but I think this is not a reasonable solution on a AVR/Arduino with limited memory.

alirezasafdari:
I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.

Width is a minimum, not a maximum. From http://www.nongnu.org/avr-libc/user-manual/group__avr__stdlib.html:

The minimum field width of the output string (including the possible '.' and the possible sign for negative values) is given in width

Try this:

void setup()
{
  Serial.begin(115200);
  Serial.println("begin");
  double numberInput = 9.0;
  char buf[50];
  dtostrf(numberInput, 5, 1, buf);
  Serial.print(buf);
}
void loop(){}

numberInput = 99999999999999999999999999999999999.0;You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.

With a bit of thought you can plan ahead when using dtostrf() and allocate a sensible amount of space. A 10 character array should suffice in reasonable situations. One character for sign and one for decimal point leaves 7 for significant figures - any more would be nonsense. Don't forget the terminating zero byte!

jremington:

numberInput = 99999999999999999999999999999999999.0;

You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.

It would be unreasonable even if a double really was twice as many bytes as a float.

I am sorry, with all due respect but this is how I look at it.

Is it unreasonable? yes.
Is it possible for the event to occur? yes.
How bad it is, if it happens? very bad.
I need a solution for this.

Specially consider the cases where the user is going to enter a number. Or even some part of the code, some how manage to produce a very large number, due to mistakes in code (debugging) or hardware error/faults (disaster).

Let's say a sensor connection come loose and the ADC reading drops to a level it would never happen if the sensor was attached to it. And for some reasons a constant is divided by this float variable coming out of sensor. Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved and then things may not look that good.

Yes, the C / C++ language affords you many opportunities to shoot yourself in the foot if you’re not paying attention - pointer manipulation being a prime example. User interaction and external hardware interfaces are always a wildcard. It’s up to the programmer to determine how much error checking, limit testing, fail safe, etc to provide. Part of that decision is assessing how disastrous the results of a failure can be.

Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved

Evidently, you need to learn some things about input conditioning and error handling, which all of us working in the real world have to deal with on a day to day basis.

It is trivial to fix or work around the limitation of dtostrf.

  1. Use snprintf() with floating point extensions enabled.

  2. Work around the limitation of dtostrf() by thinking about the problem for 10 seconds.

For example:

void setup()
{
  Serial.begin(9600);
  Serial.println("begin");
  double numberInput = 99.0;
  fp_print(numberInput);
  numberInput=999999.0;
  fp_print(numberInput);
  numberInput=10.*numberInput;
  fp_print(numberInput);
}
void loop(){}

void fp_print(float x)  //function to print a float, within range
{ 
  char buf[10];
  if ( fabs(x)<9.99999E6 ) Serial.println(dtostrf(x, 5, 2, buf));
  else Serial.println("overflow");
}

I am sorry if I am sounding arrogant but I really do not understand why you guys are trying so hard to say a problem does not exist when it does.

@jremington
For your solution 1, I do not know how the "extensions enabled" work and I could not find much information on sprintf in arduino except the fact that it does not support formatting. Will be more than happy to see a link or an elaboration of your idea.

For solution number 2, you gotta be kidding. Do you actually understand how much time it takes to check the float range for every conversion? Also, if you have ever rolled your eyes on dtostrf implementation, it knows exactly what is going to happen. it knows the exponent and can easily compute and estimate the length required right after its first step which is extracting the exponent. I have done this and considering the current attitude in the community so far, I guess I end up doing it myself and making a safe version of dtostrf (which could stop when buffer length is used up and return a false), or a conditional checking using the mathematical properties of exponent (less efficient because it will happen again in side dtostrf)

@Delta_G
I think you are focusing on the meaning of the number. Where I do not even consider that as an on topic . I understand your point, but if you have a heavy calculation in your program and some variables may happen to have an unusual values then having 35 digit number is possible. Yes, the data is in accurate and all that but it has happened and probably you do not know that it has happened in your program because you did not expected the unusual value when writing the code (yes, you can be careful and check stuff but there still might be cases where the number can get screwed up) and when the dtostrf is called it starts writing on the rest of the memory where it is not supposed to. That is not something you can forgive in most cases.

With dtostrf you get to specify how many digits to display.

I don't think that is true. If I remember correctly, you can only specify minimum size or digits. There is no maximum. I'm pretty sure I've run into what he's found and its not obvious how to shield yourself from this problem. 999--99.0 is one thing, but another is the 1.0000000009 kind of thing. That also will blow up your string buffer.

Granted I was coding a teensy when I ran into this, so maybe the Arduino itself handles this better?

-jim lee

All numbers have an infinite number of digits.

No, not the counting numbers (i.e. integers).

You are not correct regarding the counting numbers.

You are mixing oranges with your apple analogy.

Decimals do not exist in the set of integers, for example.

You are not correct.

There are things that may be counted that may not be represented in decimal form.

I’m not typing into a 64.127894 bit computer.

There isn’t even a need for a decimal when using the set of integers.

In the set of integers, 64.00 does not exist. You are comparing a number against a non-existent/undefined thing.

Just like 1.0/0 does not equal infinity.

Take a freshman level number theory class or just read the text and you’ll understand.

Not withstanding all of this, OP is really off the reservation with his/her expectations of a floating point number on a 16bit platform (even a 64 bit platform!).

For solution number 2, you gotta be kidding.

I would never kid anyone who can't be bothered to read basic documentation, and is actually surprised to learn about buffer overflow!

Honestly, I am really sad to realize that non of you try to answer the question and instead try to find off topic stuff that are not even related.

As you clearly see in the title, I have asked for a safe way. I totally understand the concerns you guys highlight with significant digits and all but that is totally unrelated to the question.

My interest is not in showing the user a huge number, my interest is in making sure if such a number for any reason was placed in the float (which is possible, even though it may be stupid or whatever) and the typical dtostrf function is called, it does not change the memory content of adjacent locations .

I seriously do not understand why so much focused is put on the meaning of the number rather than the possibility of dtosrf being called with a huge number. Also take note it can be possible that the user misses a very rare case when a number which is supposed to be one digit, become 2 digits and the buffer is only allocating enough room for one digit. the current dtostrf is going to write in another memory location which is not supposed to.

One solution is to check every float before calling the dtostrf but as I have mentioned before that would take so much time since you have to make sure each number is within the positive and negative boundary. That is 2 float comparison for each time dtostrf is called and the boundary changes
based on formatting. I really do not consider this a solution, if you do; then you may just stick to this solution and hopefully you will have enough time for everything else in the project.

Yeah, I may have misread the documentation but that is because the dtostrf is very unsafe and I had higher expectation from the general dtostrf. This implementation is ok for when you have an operating system or something to stop you from screwing the memory but in arduino we clearly do not have such a guard and you can try all you want talking about significant digit and all the off topic stuff you have written here but you do not change the fact that this function is very unsafe and making it safe is actually very very easy. As I stated I will do it myself and share the code here withing next few days, so probably folks who understand code safety can be benefited.

After all if you have suggestion on topic, I will be more than happy to hear it.

Delta_G Please read the last part first.

"how that huge number is represented and how 1.0 is represented"
You are wrong!

{
float magicNumber = 1.0;
....
float a = magicNumber;
char buffer[3];
dtostrf(a, 0, 1, buffer);
}

[1]- [0]
works perfectly fine but

{
float magicNumber = 1000.0;
....
float a =magicNumber;
char buffer[3];
dtostrf(a, 0, 1, buffer);
}

[1][0][0][0]- [0]
is a disaster. I hope you understand that.

Yes this could be avoided if we could know the exact value that comes out of magicNumber. But this magicNumber may come from another unit and if that unit fails to stay in range, you have to make sure the arduino side does not explode = writing in memory location that does not belong to the buffer.

"They're both going to be 32bit floating point which has about 6 or 7 digits of precision."
I do not know how is this even related. Yes all floats and doubles take 32 bit in Arduino but does it mean when they are printed by dtostrf with zero digit after the decimal point, the take same amount of memory? definitely not. (I hope you understand this too)

"When you call dtostrf you get to specify the precision so you get to specify how many digits get printed. "
You have probably read the link below but this is not how the arduino dtostrf work. You have probably read "prec determines the number of digits after the decimal sign" but in arduino it is "prec determines the number of digits after the decimal point". You can give it a shot if you do not believe me.

I think this is the reason why there is confusion here. I also checked the stdlib.h and the description is wrong.

I do not think the rest of the sprintf over head is not worth it nor needed.

That does seem a bit annoying, doesn't it?
How about dtostre(), which prints in the "+1.2345e-06" format? You still get to specify "precision" for number of places after the decimal point, and that should yield a known maximum total width...

alirezasafdari:
Hi

I have been using in dtosrtf() in a relatively big project and some usual stuff was happening. Long story short, I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory.

Someone in another forum suggested using 100 byte array to be safe, but I think this is not a reasonable solution on a AVR/Arduino with limited memory.

So sad that the Arduino IDE developers refuse to let users do something like this:

int main (void)
{
    init();
    Serial.begin (115200);

    STDIO.open (Serial);

    double a = 123.45678;
    double b = -0.0045;
    double c = 1.2345678;

    fprintf (stdout, "The variable 'a' has a value of %7.3f\n", a);
    fprintf (stdout, "The variable 'b' has a value of %7.3f\n", b);
    fprintf (stdout, "The variable 'b' has a value of %7.3f\n", c);

    while (1);
}

Output from this code:

[b]The variable 'a' has a value of 123.457
The variable 'b' has a value of  -0.004
The variable 'b' has a value of   1.235
[/b]

See? No "dtostrf" baloney.

Delta_G:
Is it really the IDE that stops you? I thought it was GCC that was the issue there. And they don't refuse to let you. You did it. It's open source, you can do anything you want with it.

GCC does everything just fine. All the IDE does is build GCC command strings and passes them to the compiler. Any limitations in programming an Arduino are due to the IDE, NOT GCC.

To use ordinary printf and floating point, all that's necessary is to "hack" the IDE to link [b]libprintf_flt.a[/b] into the compiled binary instead of [b]libprintf_min.a[/b].

Or, better yet, add a checkbox option in Preferences to choose which one to link (floating point when you need it, the non-floating point one when you don't - saves a few bytes of flash).

The "sketch" I show above was compiled by a plain old Arduino 1.0.5 IDE with a few minor differences:

(1) I turned on the floating point option checkbox (custom option in my IDE).
(2) I connected the serial port to the standard IO streams to allow using fprintf (the STDIO.open() call).
(3) I used "int main (void)" instead of that "setup/loop" nonsense (although it would work the same with setup/loop).

See? I didn't need "dtostrf". I didn't need to worry about buffer overflows. I didn't need a dozen "Serial.print()" calls just to print a few lines, I didn't need to play games checking string lengths and inserting padding to get the numbers to line up right.

@westfw that could be a solution but honestly showing the scientific notations may not be the best way to present data mainly it is harder to read and understand if the numbers are small most of the time. So, I am still going to work on a solution which safely perform this. Please read the rest of my post after replies to each individual, because I have a question which probably you can answer it the best.

@krupski I am not really familiar with IDE manipulations and most stuff in that regard which you mentioned. I think the method you proposed has a lot of over head in run time. but a "safe, civilized, reasonable" dtostrf is really easy to do and also consider that we may not always need to put things in serial port (although I know sprintf could be used if some tweaks are done to IDE).

Now back to question:

  1. so is it concluded that the description for dtostrf is wrong or dtostrf is implemented wrong? if yes, could you guys with higher star rating and reputation voice this out so that it can get fixed. Or if I should report it myself please let me know what is the best way to do it.

  2. I was checking the implementation of dtostrf from this (thank to westfw) and I noticed there is an interesting function which does most of the job and probably is written in the most efficient way. that function/macro looking function is __ftoa_engine.
    A good documentation can be found in here . Before finding this document I noticed that __ftoa_engine cannot be used in arduino IDE. and I could not even include ftoa_engine.h. How can I get access to this file in arduino? should not it be part of the IDE?

  3. In the reference above for avr-libc.1.8.0 a dtostrf implementation has been provided which follows the exact documention as the one used in arduino but it has been implemented correctly, which means the user can tell the precision."prec determines the number of digits after the decimal sign". What do you think about this implementation?