Arduino Forum

Using Arduino => Programming Questions => Topic started by: alirezasafdari on Nov 08, 2017, 09:47 am

Title: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 08, 2017, 09:47 am
Hi

I have been using in dtosrtf() in a relatively big project and some usual stuff was happening. Long story short, I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory.

This is very surprising for me and for those who may have a hard time believing it, I have written a little code here.

Code: [Select]
volatile float numberInput = -3, zero = 0;
char buf[50];

void setup()
{
Serial.begin(115200);
Serial.println("begin");

numberInput = 99999999999999999999999999999999999.0;
dtostrf(numberInput, 5, 1, buf);
// expectation: it will not go beyond 5 character in the buffer
// reality: it goes and screw all the data in the memmory
// question: how do we do this safely?
Serial.write(buf, 50);
Serial.println();
}


And the result in serial port was shocking

Code: [Select]
100000000000000000000000000000000000.0..............
The . represent 0x00. (checked with non Arduino serial monitor). No surprise there.


And just to show how dangerous it can get, the following code it keeps printing "begi" in serial port. (it continuously start from the start of the program) (I just changed buffer size to 5 instead of 50)

Code: [Select]
volatile float numberInput = -3, zero = 0;
char buf[5];

void setup()
{
Serial.begin(115200);
Serial.println("begin");

numberInput = 99999999999999999999999999999999999.0;
dtostrf(numberInput, 5, 1, buf);
// expectation: it will not go beyond 5 character in the buffer
// reality: it goes and screw all the data in the memmory
// question: how do we do this safely?
Serial.write(buf, 50);
Serial.println();
}





When I was suspicious of dtostrf(), I taught looking at the implementation would help (I asked in forum (http://forum.arduino.cc/index.php?topic=510159.0)). However, I was very wrong but in case that helps you to come up with a solution, you can check the implementation here:

http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/dtoa_prf.c?revision=1944&view=markup (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/dtoa_prf.c?revision=1944&view=markup)

By safe and reasonable, I am looking for a solution where I do not have to worry if I do not provide enough space. I prefer to get acknowledged that the operation was not successful rather than the whole memory being screwed. BTW I am not asking for a magical solution, so I let the function know how much space it has in the buffer, so that it does not go all over the place.

Regarding dtosrtf: I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.

Someone in another forum suggested using 100 byte array to be safe, but I think this is not a reasonable solution on a AVR/Arduino with limited memory.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: pert on Nov 08, 2017, 09:58 am
I initially taught the width is in charge of telling the limits and right now I do not even know what the width does.
Width is a minimum, not a maximum. From http://www.nongnu.org/avr-libc/user-manual/group__avr__stdlib.html (http://www.nongnu.org/avr-libc/user-manual/group__avr__stdlib.html):
Quote
The minimum field width of the output string (including the possible '.' and the possible sign for negative values) is given in width
Try this:
Code: [Select]
void setup()
{
  Serial.begin(115200);
  Serial.println("begin");
  double numberInput = 9.0;
  char buf[50];
  dtostrf(numberInput, 5, 1, buf);
  Serial.print(buf);
}
void loop(){}
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: jremington on Nov 08, 2017, 04:25 pm
Code: [Select]
numberInput = 99999999999999999999999999999999999.0;You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.

With a bit of thought you can plan ahead when using dtostrf() and allocate a sensible amount of space. A 10 character array should suffice in reasonable situations. One character for sign and one for decimal point leaves 7 for significant figures - any more would be nonsense. Don't forget the terminating zero byte!
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: gfvalvo on Nov 08, 2017, 04:41 pm
Code: [Select]
numberInput = 99999999999999999999999999999999999.0;
You should know that floats and doubles are the same on Arduino, and are accurate to only 6-7 digits. So, this number is unreasonable in terms of significant digits.
It would be unreasonable even if a double really was twice as many bytes as a float.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 08, 2017, 09:03 pm
I am sorry, with all due respect but this is how I look at it.

Is it unreasonable? yes.
Is it possible for the event to occur? yes.
How bad it is, if it happens? very bad.
I need a solution for this.

Specially consider the cases where the user is going to enter a number. Or even some part of the code, some how manage to produce a very large number, due to mistakes in code (debugging) or hardware error/faults (disaster).

Let's say a sensor connection come loose and the ADC reading drops to a level it would never happen if the sensor was attached to it. And for some reasons a constant is divided by this float variable coming out of sensor. Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved and then things may not look that good.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: gfvalvo on Nov 08, 2017, 09:21 pm
Yes, the C / C++ language affords you many opportunities to shoot yourself in the foot if you're not paying attention - pointer manipulation being a prime example. User interaction and external hardware interfaces are always a wildcard. It's up to the programmer to determine how much error checking, limit testing, fail safe, etc to provide. Part of that decision is assessing how disastrous the results of a failure can be.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: jremington on Nov 08, 2017, 10:46 pm
Quote
Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved
Evidently, you need to learn some things about input conditioning and error handling, which all of us working in the real world have to deal with on a day to day basis.

It is trivial to fix or work around the limitation of dtostrf.

1. Use snprintf() with floating point extensions enabled.

2. Work around the limitation of dtostrf() by thinking about the problem for 10 seconds.

For example:

Code: [Select]
void setup()
{
  Serial.begin(9600);
  Serial.println("begin");
  double numberInput = 99.0;
  fp_print(numberInput);
  numberInput=999999.0;
  fp_print(numberInput);
  numberInput=10.*numberInput;
  fp_print(numberInput);
}
void loop(){}

void fp_print(float x)  //function to print a float, within range
{
  char buf[10];
  if ( fabs(x)<9.99999E6 ) Serial.println(dtostrf(x, 5, 2, buf));
  else Serial.println("overflow");
}
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:22 am
Let's say a sensor connection come loose and the ADC reading drops to a level it would never happen if the sensor was attached to it. And for some reasons a constant is divided by this float variable coming out of sensor. Then your dtostrf is going to mess the whole memory. Put it in a expensive project with mechanical stuff involved and then things may not look that good.
You are misunderstanding what a float or double variable can or cannot do.  Even if the number gets as large as 35 9's in a row, a float variable can't represent that exactly.  It's only going to be able to do 6 or maybe 7 digits of precision.  So you wouldn't want to print out all 35 digits.  They wouldn't be right anyway.  Just print out the 6 digits you know are good.  So represent it as 9.99999E34.  With dtostrf you get to specify how many digits to display.  So you'll know the maximum possible width of the string it produces and can plan your array accordingly.  Trying to capture all those digits is just a total waste of your time and effort. 
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 09, 2017, 05:30 am
I am sorry if I am sounding arrogant but I really do not understand why you guys are trying so hard to say a problem does not exist when it does.

@jremington
For your solution 1, I do not know how the "extensions enabled" work and I could not find much information on sprintf in arduino except the fact that it does not support formatting. Will be more than happy to see a link or an elaboration of your idea.

For solution number 2, you gotta be kidding. Do you actually understand how much time it takes to check the float range for every conversion? Also, if you have ever rolled your eyes on dtostrf implementation, it knows exactly what is going to happen. it knows the exponent and can easily compute and estimate the length required right after its first step which is extracting the exponent. I have done this and considering the current attitude in the community so far, I guess I end up doing it myself and making a safe version of dtostrf (which could stop when buffer length is used up and return a false), or a conditional checking using the mathematical properties of exponent (less efficient because it will happen again in side dtostrf)


@Delta_G
I think you are focusing on the meaning of the number. Where I do not even consider that as an on topic . I understand your point, but if you have a heavy calculation in your program and some variables may happen to have an unusual values then having 35 digit number is possible. Yes, the data is in accurate and all that but it has happened and probably you do not know that it has happened in your program because you did not expected the unusual value when writing the code (yes, you can be careful and check stuff but there still might be cases where the number can get screwed up) and when the dtostrf is called it starts writing on the rest of the memory where it is not supposed to. That is not something you can forgive in most cases.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: jimlee2 on Nov 09, 2017, 09:16 am
Quote
With dtostrf you get to specify how many digits to display.
I don't think that is true. If I remember correctly, you can only specify minimum size or digits. There is no maximum. I'm pretty sure I've run into what he's found and its not obvious how to shield yourself from this problem. 999--99.0 is one thing, but another is the 1.0000000009 kind of thing. That also will blow up your string buffer.

Granted I was coding a teensy when I ran into this, so maybe the Arduino itself handles this better?

-jim lee
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 02:33 pm
No, apparently you don't understand how numbers work. There are no such thing as numbers that have more or less digits than other numbers. All numbers have an infinite number of digits. If you say 1 then I can say 1.0 and someone else can say 1.00.  Whether or not you write all the digits when you write the number back to the human doesn't matter.  The math isn't being done with the output of dtostrf.  The math is being done with the actual floating point number.  So you're not losing on your math.  Your math in your program is safe. The only thing that String is for is showing the human what the value is.  And if you try to show him 35 digits then you are lying to your user about the value.

If the result is some number with 35 digits why in the world would a user want you to output that as a 35 digit number with only 6 that matter and 29 that are basically just made up when you can give him scientific notation and only the part that's accurate and actually conveys the value. 

This problem isn't unique to computers.  Big numbers come up in science all the time and with propagation of error from measurements we know how many significant digits we actually have. And we don't ever write out 35 digit numbers there.  At least not unless we actually have 35 digits of precision. 

Sure, big numbers might come up and they come up all the time.  But that doesn't mean it makes sense to write them all the way out without scientific notation. 
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 09, 2017, 02:48 pm
Quote
All numbers have an infinite number of digits.
No, not the counting numbers (i.e. integers).
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:09 pm
No, not the counting numbers (i.e. integers).
Yes, even them.  You have to separate the abstract concept of a number, from the concrete representation of the number.  For the abstract concept pretend I have 3 apples.  Would you write that I have 3, 3.0, 3.00, 3.000, 3.0000.   How many apples do I have there? 

Now the concrete is different.  If someone wrote that I had 3 apples, you aren't entirely sure I didn't have a little piece of an extra apple or that I hadn't taken a bite out of one of them and the writer rounded off.  But if he said that I had 3.0 apples, you could be sure that I hadn't bitten off more than a tenth of one.  If he said I had 3.00 apples then you're sure I haven't bitten off more than a hundredth of one. 

The key concept that the OP needs to grasp is the concept of significant digits.  Many a student has messed up their grade in chemistry class over sig figs. 

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 09, 2017, 03:14 pm
You are not correct regarding the counting numbers.

You are mixing oranges with your apple analogy.

Decimals do not exist in the set of integers, for example.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:18 pm
You are not correct regarding the counting numbers.

You are mixing oranges with your apple analogy.

Decimals do not exist in the set of integers, for example.
You're talking about representations of quantities. 

I am talking about numbers in the abstract.  Not the representation of quantity but the actual quantity itself.  I don't mean the part that you read and write, but the part that you actually do math with.   

If I have 3 apples and you have 3.0 apples, sure you represented yours with an integer, but we both have the same actual quantity of apples.  If we traded, it would still be an even trade.  Unless, of course, you were rounding. 
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 09, 2017, 03:28 pm
You are not correct.

There are things that may be counted that may not be represented in decimal form. 

I'm not typing into a 64.127894 bit computer.

There isn't even a need for a decimal when using the set of integers.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:36 pm
There isn't even a need for a decimal when using the set of integers.


There isn't a need sure, but that doesn't mean that the number isn't that exact.  Again, you're talking about how you would represent that number.  Whether or not you should WRITE a decimal point.  But the number itself is the same number even if you do. 

Are you telling me that 64 != 64.00?
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 09, 2017, 03:44 pm
In the set of integers, 64.00 does not exist. You are comparing a number against a non-existent/undefined thing.

Just like 1.0/0 does not equal infinity.

Take a freshman level number theory class or just read the text and you'll understand.

Not withstanding all of this, OP is really off the reservation with his/her expectations of a floating point number on a 16bit platform (even a 64 bit platform!).
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:48 pm
In the science world we actually do treat the counting numbers differently from measured numbers.  So if we measure with a ruler 3.1 cm then we say that number has 2 significant figures and when you do math with it you have to count those two sig figs and no more.  

Now when we are counting that is different from measuring.  If I measure with a ruler then I can say it was real close to the 3 line, but I can never say it was exactly that.  We could always zoom in more until we have a microscope that can see the Planck length.  

But when we are counting we know exactly.  That isn't like measuring.  If I count 3 cows in the field, looking closer isn't going to reveal 0.001 cows that we just couldn't see before.  So when you're doing calculations in chemistry class and you're counting your sig figs, you actually count the counting numbers as infinite accuracy to keep them from interfering with the number of sig figs coming from the measured numbers.  So in that treatment you are pretending that the integer has an infinite number of digits.  Not that you would ever write it that way, but for the purposes of the calculation you treat it as that exact.  
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 09, 2017, 03:52 pm
Take a freshman level number theory class or just read the text and you'll understand.
I have.  And I'm talking about quantity, not number set theory.  To say that I have a certain quantity does not limit me to any particular set of numbers narrower than the set of real numbers.  Perhaps I should concede that I should have used the word quantity instead of number in the original line that set this off. 

Either way, I think we're both arguing true points about two completely different things. 

Here's another point for the OP if he is still with us.  We work with a number all the time that has a whole bunch of digits.  Think about pi.  If you are working with pi, do you need to make sure you capture ALL of the digits (an impossibility) or do you round it at some point where it is accurate enough for your particular needs?
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: jremington on Nov 09, 2017, 05:18 pm
Quote
For solution number 2, you gotta be kidding.
I would never kid anyone who can't be bothered to read basic documentation, and is actually surprised to learn about buffer overflow!
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 10, 2017, 12:54 am
Honestly, I am really sad to realize that non of you try to answer the question and instead try to find off topic stuff that are not even related.

As you clearly see in the title, I have asked for a safe way. I totally understand the concerns you guys highlight with significant digits and all but that is totally unrelated to the question.

My interest is not in showing the user a huge number, my interest is in making sure if such a number for any reason was placed in the float (which is possible, even though it may be stupid or whatever) and the typical dtostrf function is called, it does not change the memory content of adjacent locations .

I seriously do not understand why so much focused is put on the meaning of the number rather than the possibility of dtosrf being called with a huge number. Also take note it can be possible that the user misses a very rare case when a number which is supposed to be one digit, become 2 digits and the buffer is only allocating enough room for one digit. the current dtostrf is going to write in another memory location which is not supposed to.

One solution is to check every float before calling the dtostrf but as I have mentioned before that would take so much time since you have to make sure each number is within the positive and negative boundary. That is 2 float comparison for each time dtostrf is called and the boundary changes
 based on formatting. I really do not consider this a solution, if you do; then you may just stick to this solution and hopefully you will have enough time for everything else in the project.

Yeah, I may have misread the documentation but that is because the dtostrf is very unsafe and I had higher expectation from the general dtostrf. This implementation is ok for when you have an operating system or something to stop you from screwing the memory but in arduino we clearly do not have such a guard and you can try all you want talking about significant digit and all the off topic stuff you have written here but you do not change the fact that this function is very unsafe and making it safe is actually very very easy. As I stated I will do it myself and share the code here withing next few days, so probably folks who understand code safety can be benefited.

After all if you have suggestion on topic, I will be more than happy to hear it.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 10, 2017, 01:04 am
The thing you don't understand is that there's no difference in how that huge number is represented and how 1.0 is represented insofar as number of digits is concerned.  They're both going to be 32bit floating point which has about 6 or 7 digits of precision.  When you call dtostrf you get to specify the precision so you get to specify how many digits get printed. 

If you're really worried about it, re-enable floating point support for format strings and use snprintf
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 10, 2017, 02:39 am
Delta_G Please read the last part first.

"how that huge number is represented and how 1.0 is represented"
You are wrong!
Code: [Select]

{
float magicNumber = 1.0;
....
float a = magicNumber;
char buffer[3];
dtostrf(a, 0, 1, buffer);
}

[1][.][0]
works perfectly fine but
Code: [Select]

{
float magicNumber = 1000.0;
....
float a =magicNumber;
char buffer[3];
dtostrf(a, 0, 1, buffer);
}

[1][0][0][0][.][0]
is a disaster. I hope you understand that.

Yes this could be avoided if we could know the exact value that comes out of magicNumber. But this magicNumber may come from another unit and if that unit fails to stay in range, you have to make sure the arduino side does not explode = writing in memory location that does not belong to the buffer.


"They're both going to be 32bit floating point which has about 6 or 7 digits of precision."
I do not know how is this even related. Yes all floats and doubles take 32 bit in Arduino but does it mean when they are printed by dtostrf with zero digit after the decimal point, the take same amount of memory? definitely not. (I hope you understand this too)


"When you call dtostrf you get to specify the precision so you get to specify how many digits get printed. "
You have probably read the link below but this is not how the arduino dtostrf work. You have probably read "prec determines the number of digits after the decimal sign" but in arduino it is "prec determines the number of digits after the decimal point". You can give it a shot if you do not believe me.

http://www.atmel.com/webdoc/avrlibcreferencemanual/group__avr__stdlib_1ga060c998e77fb5fc0d3168b3ce8771d42.html (http://www.atmel.com/webdoc/avrlibcreferencemanual/group__avr__stdlib_1ga060c998e77fb5fc0d3168b3ce8771d42.html)

I think this is the reason why there is confusion here. I also checked the stdlib.h and the description is wrong.


I do not think the rest of the sprintf over head is not worth it nor needed.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: westfw on Nov 10, 2017, 05:31 am
That does seem a bit annoying, doesn't it?
How about dtostre(), which prints in the "+1.2345e-06" format?  You still get to specify "precision" for number of places after the decimal point, and that should yield a known maximum total width...

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 10, 2017, 05:53 am
That's the one I was looking for.  I knew you could get it out in scientific notation.  That's what you do with big numbers.  dtostre
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 10, 2017, 06:31 am
Hi

I have been using in dtosrtf() in a relatively big project and some usual stuff was happening. Long story short, I took "The caller is responsible for providing sufficient storage in s" not as serious as I should. I thought if I do not provide enough width it automatically understand not to write in the rest of the memory.

Someone in another forum suggested using 100 byte array to be safe, but I think this is not a reasonable solution on a AVR/Arduino with limited memory.
So sad that the Arduino IDE developers refuse to let users do something like this:

Code: [Select]
int main (void)
{
    init();
    Serial.begin (115200);

    STDIO.open (Serial);

    double a = 123.45678;
    double b = -0.0045;
    double c = 1.2345678;

    fprintf (stdout, "The variable 'a' has a value of %7.3f\n", a);
    fprintf (stdout, "The variable 'b' has a value of %7.3f\n", b);
    fprintf (stdout, "The variable 'b' has a value of %7.3f\n", c);

    while (1);
}



Output from this code:

The variable 'a' has a value of 123.457
The variable 'b' has a value of  -0.004
The variable 'b' has a value of   1.235

See? No "dtostrf" baloney.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Delta_G on Nov 10, 2017, 06:34 am
So sad that the Arduino IDE developers refuse to let users do something like this:

Is it really the IDE that stops you?  I thought it was GCC that was the issue there.  And they don't refuse to let you.  You did it.  It's open source, you can do anything you want with it. 
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 10, 2017, 06:52 am
Is it really the IDE that stops you?  I thought it was GCC that was the issue there.  And they don't refuse to let you.  You did it.  It's open source, you can do anything you want with it.  
GCC does everything just fine. All the IDE does is build GCC command strings and passes them to the compiler. Any limitations in programming an Arduino are due to the IDE, NOT GCC.

To use ordinary printf and floating point, all that's necessary is to "hack" the IDE to link  libprintf_flt.a  into the compiled binary instead of  libprintf_min.a.

Or, better yet, add a checkbox option in Preferences to choose which one to link (floating point when you need it, the non-floating point one when you don't - saves a few bytes of flash).

The "sketch" I show above was compiled by a plain old Arduino 1.0.5 IDE with a few minor differences:

(1) I turned on the floating point option checkbox (custom option in my IDE).
(2) I connected the serial port to the standard IO streams to allow using fprintf (the STDIO.open() call).
(3) I used "int main (void)" instead of that "setup/loop" nonsense (although it would work the same with setup/loop).

See? I didn't need "dtostrf". I didn't need to worry about buffer overflows. I didn't need a dozen "Serial.print()" calls just to print a few lines, I didn't need to play games checking string lengths and inserting padding to get the numbers to line up right.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 10, 2017, 11:25 pm
@westfw that could be a solution but honestly showing the scientific notations may not be the best way to present data mainly it is harder to read and understand if the numbers are small most of the time. So, I am still going to work on a solution which safely perform this. Please read the rest of my post after replies to each individual, because I have a question which probably you can answer it the best.

@krupski I am not really familiar with IDE manipulations and most stuff in that regard which you mentioned. I think the method you proposed has a lot of over head in run time. but a "safe, civilized, reasonable" dtostrf is really easy to do and also consider that we may not always need to put things in serial port (although I know sprintf could be used if some tweaks are done to IDE).



Now back to question:

1) so is it concluded that the description for dtostrf is wrong or dtostrf is implemented wrong? if yes, could you guys with higher star rating and reputation voice this out so that it can get fixed. Or if I should report it myself please let me know what is the best way to do it.


2) I was checking the implementation of dtostrf from this (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/dtoa_prf.c?revision=1944&view=markup) (thank to westfw) and I noticed there is an interesting function which does most of the job and probably is written in the most efficient way. that function/macro looking function is __ftoa_engine.
A good documentation can be found in here  (https://sourcecodebrowser.com/avr-libc/1.8.0/ftoa__engine_8h.html). Before finding this document I noticed that __ftoa_engine cannot be used in arduino IDE. and I could not even include ftoa_engine.h. How can I get access to this file in arduino? should not it be part of the IDE?


3) In the reference above for avr-libc.1.8.0 a dtostrf implementation has been provided which follows the exact documention as the one used in arduino but it has been implemented correctly, which means the user can tell the precision."prec determines the number of digits after the decimal sign". What do you think about this implementation?
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 11, 2017, 01:01 am
@krupski I am not really familiar with IDE manipulations and most stuff in that regard which you mentioned. I think the method you proposed has a lot of over head in run time.
I'm confused. You are "not really familiar" with how all this works, yet you assert that using the compiler properly creates a "lot of overhead in run time".

Now, I know that a lot of people feel that enabling floating point support uses a lot of extra flash memory. And, indeed it does use more flash, but not THAT much. I have yet to come anywhere close to filling all the flash.

Another thing which most people don't realize is that "floating point support" entails TWO DIFFERENT pieces of code that are linked into the program at compile time.

They are:

The first one supports all of the printf related functions. The second one supports scanf, sscanf, etc... (functions that are rarely used).

Both of these utilize extra flash (and a bit of ram as well) space.  The second one (scanf) uses the bulk of resources and is rarely if ever used.

The printf one, however, uses very little memory and therefore there is NO REASON to not use it. You can freely enable one, the other or both - doesn't matter.

If you want to try it without hacking anything, do this:
Go to your "arduino_dir/hardware/tools/avr/avr/lib/" directory.

Rename the file "libprintf_min.a" to "libprintf_min.a.backup".
Copy the file "libprintf_flt.a" to "libprintf_min.a"

Now, write a small sketch like this:
Code: [Select]
void setup (void)
{
    char buffer [64];
    double value = 123.45678;
    Serial.begin (9600); // or whatever baud rate you want
    sprintf (buffer, "You should see a number here --> %7.3f\r\n", value);
    Serial.print (buffer);
}

void loop (void)
{
    /* nothing */
}



Do you see what we did here?

We replaced the "minimum" printf code with the "floating point" printf code.  Since you won't have printf enabled, you can test it by using sprintf to a buffer, then printing the buffer.

If it works, you should see this:  "You should see a number here --> 123.457"
It not, you will get this:  "You should see a number here -->       ?"

To un-do the "hack", just rename those two files above back to what they were.

In fact, FIRST try the sketch above before doing the mod and see how much ram and flash is used, then do the mod and compare resource usage. You will see that it's a trivial amount.

Good luck
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 11, 2017, 03:34 am
Sorry krupski. I just stated what I have read and did not test it myself. I give it a shot now. Just one question, does the new files use heap? I am going to test it anyway but just wanted to know if they use heap.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: westfw on Nov 11, 2017, 04:14 am
Quote
It is trivial to fix or work around the limitation of dtostrf.
1. Use snprintf() with floating point extensions enabled.
I think that is "far from trivial" by Arduino standards, unfortunately.
Quote
is it concluded that the description for dtostrf is wrong or dtostrf is implemented wrong?
I think it matches the (avr-libc documentation) description.  "width" was documented as "minimum width" and precision is "number of places after decimal."  There does not seem to be a "maximum width" parameter, so numbers like "99e20" are a problem.
The floating point format (on AVR) is limited to values <1039, so a number will never take more than sign+39digits+.+<prec> bytes, I think...

It's worth remembering why printf() and floatingPoint printf() are not the defaults.  a printf() based implementation of "hello world" is about 1k bigger than the same with Serial.print(), and using the floating point version of printf() expands the difference to about 4kbytes, WHETHER OR NOT YOU USE THE FLOATING POINT FEATURES.  Those were big numbers back when a "large" AVR only had 4k to 8k of code space.
(OTOH, the difference in size between using the floating point printf vs using the Arduino Serial.print(float,prec) is only a couple hundred bytes.)
I'm a little disappointed that the Arduino team keeps extending Serial.print() instead of embracing industry standards.
OTOH, I'm also a bit disappointed that dtostr() isn't just a wrapper for __ftoa_engine() or something; it's not clear to me how they're different.   And this (in the dtoa_prf.c) seems a bit worrisome:
   unsigned char buf[9];
       :
    ndigs = prec < 60 ? prec + 1 : 60;
    exp = __ftoa_engine (val, (char *)buf, 7, ndigs);



Looks to me like they just told the engine that their 9byte buffer had room for 60 digits...
And ... It looks like the ftoa_engine output is ... really weird.



Quote
One solution is to check every float before calling the dtostrf but as I have mentioned before that would take so much time since you have to make sure each number is within the positive and negative boundary. That is 2 float comparison for each time dtostrf is called
Actually, floating point comparisons are very cheap.   MUCH cheaper than the first division that WILL happen if you actually do the conversion.  (in fact, most FP formats are cleverly designed so that a floating point comparison can happen nearly as quickly as a fixed point comparison (even without FP hardware.)  Except for NaN and other "features."  That's why you see things like "excess128" instead of two's complement used for the exponent.)
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: Jimmus on Nov 11, 2017, 08:02 am
There is a limit on how big your float can be.  The documentation says it's -3.4028235E+38.  I can confirm that if you try to print out a number bigger than that (like -3.4028236E+38), it turns it into "INF" or "-INF".

So the solution to your problem is to supply a buffer that is 43 characters or larger.  Because yeah, if you only supply 42, it could crash.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 11, 2017, 08:41 am
Sorry krupski. I just stated what I have read and did not test it myself. I give it a shot now. Just one question, does the new files use heap? I am going to test it anyway but just wanted to know if they use heap.
They are not "new" files, you are simply tricking GCC into linking in the floating point code when it's actually trying to link in the "minimal" (non-floating point) code.

As far as "heap", I don't really know or care how GCC allocates memory. I only care that I have ENOUGH.

That's why I suggested to build the test program first, then see how much flash and ram it uses, then do the file rename trick to use the floating point code and build the exact same test program again and compare flash and ram usage.

If you like how it works and you feel the little extra memory usage is OK, then stick with it. If not, simply rename the files back to original and all is back to the way it was before.


Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 11, 2017, 09:04 am
It's worth remembering why printf() and floatingPoint printf() are not the defaults.  a printf() based implementation of "hello world" is about 1k bigger than the same with Serial.print(), and using the floating point version of printf() expands the difference to about 4kbytes, WHETHER OR NOT YOU USE THE FLOATING POINT FEATURES.  Those were big numbers back when a "large" AVR only had 4k to 8k of code space.
Did you see my post (http://forum.arduino.cc/index.php?topic=510327.msg3482097#msg3482097) a few places up?

There are TWO different floating point support files... one for printf and related, another for scanf and related.

There is no need to link in BOTH of them, You can use one, the other or both.  And, since scanf and related are rarely used, one can link in ONLY the printf floating point code and save quite a bit of space.

That's why I have the option to chose one, the other or both in my Preferences.

(click image for full size)
(http://forum.arduino.cc/index.php?action=dlattach;topic=510327.0;attach=233008) (http://forum.arduino.cc/index.php?action=dlattach;topic=510327.0;attach=233008)


Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 11, 2017, 10:48 am
@krupski

I have tried your trick by renaming the files and it did not work. I think it was simple enough for me not to make a mistake but again I could be wrong. (I am using latest Arduino).

However since I did a small test, I decided to share the result (not sure if it is meaningful but again it is a job done)

For Dtostrf:
Code: [Select]
#include "MemoryFree.h"

void setup()
{
 int availableMemory;
 availableMemory = freeMemory();

 Serial.begin(115200);
 uint8_t buffer[60] = {};
 float inputNumber = 1357.125;
 uint32_t start, end;
 
 Serial.println(availableMemory);
 
 start = micros();
 dtostrf(inputNumber, 0, 3, buffer);
 end = micros();
 
 Serial.println(end - start);

 Serial.write(buffer, 60);
 
}
// the loop function runs over and over again forever
void loop()
{

 
}


Result:
Quote
Using dtostrf:
Flash space = 3790
Global variable = 190
Free memory at the beginning = 7937
Micros taken = 120
Output is: 1357.125 [00] X 52
for sprintf:
Code: [Select]
#include "MemoryFree.h"

void setup()
{
 int availableMemory;
 availableMemory = freeMemory();

 Serial.begin(115200);
 uint8_t buffer[60] = {};
 float inputNumber = 1357.125;
 uint32_t start, end;
 
 Serial.println(availableMemory);
 
 start = micros();
 sprintf (buffer, "%7.3f\r", inputNumber);
 end = micros();
 
 Serial.println(end - start);

 Serial.write(buffer, 60);
 
}

// the loop function runs over and over again forever
void loop()
{

 
}


results:
Quote
Without modifying the IDE:
Flash space = 3766
Global variable = 196
Free memory at the beginning = 7931
Micros taken = 80
Output is: [20] [20] [20] [20] [20] [20][3F = '?'][0D = '\n'] [00] X 52

With modifying the IDE:
Flash space = 3766
Global variable = 196
Free memory at the beginning = 7931
Micros taken = 80
Output is: [20] [20] [20] [20] [20] [20][3F = '?'][0D = '\n'] [00] X 52
@westfw
I am not sure about (avr-libc documentation) but the the other stostrf I found they let you set the number of digits after the sign. This technically solve all the problems. I am not sure what they do if they cannot fit it in the given number of digits but they definitely have a way to know.

__ftoa_engine() precision is different from in current arduino dtostrf. the precision tells the maximum number of digit coming out. that is why they put a 7 there to indicate the want only 7 numbers to be printed in a 9 byte buffer. Not sure about the return though. The implementation is odd with zero comments which make it tougher.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: oqibidipo on Nov 11, 2017, 01:32 pm
Rename the file "libprintf_min.a" to "libprintf_min.a.backup".
Copy the file "libprintf_flt.a" to "libprintf_min.a"

libprintf_min.a is not the default printf, but an even more stripped down version.
The default version is in libc.a.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 13, 2017, 01:06 am
libprintf_min.a is not the default printf, but an even more stripped down version.
The default version is in libc.a.
Here is the exact piece of code that I have in "Compiler.java" to do the job:

Code: [Select]
///////////////// this provides the floating point option /////////////////

if (Preferences.getBoolean ("build.printf_floating_point")) {
    baseCommandLinker.add ("-Wl,-u,vfprintf,-lprintf_flt");
}

if (Preferences.getBoolean ("build.scanf_floating_point")) {
    baseCommandLinker.add ("-Wl,-u,vfscanf,-lscanf_flt");
}

///////////////// this provides the floating point option /////////////////


...and this code provides the check boxes in "Preferences,java" to select or deselect either one:

Code: [Select]
// [ ] printf Floating point support
useFFloatingPointBox = new JCheckBox (("  Enable (f)printf floating point"));
useFFloatingPointBox.setToolTipText (getToolTip ("Enable floating point support for printf"));
pane.add (useFFloatingPointBox);
d = useFFloatingPointBox.getPreferredSize();
useFFloatingPointBox.setBounds (left, top, d.width + 10, d.height);
right = Math.max (right, left + d.width);
top += d.height + GUI_BETWEEN;

// [ ] scanf Floating point support
useSFloatingPointBox = new JCheckBox (("  Enable (f)scanf floating point"));
useSFloatingPointBox.setToolTipText (getToolTip ("Enable floating point support for scanf"));
pane.add (useSFloatingPointBox);
d = useSFloatingPointBox.getPreferredSize();
useSFloatingPointBox.setBounds (left, top, d.width + 10, d.height);
right = Math.max (right, left + d.width);
top += d.height + GUI_BETWEEN;



And finally this code reads and writes the preferences in "Preferences.txt":

Code: [Select]
// in "applyFrame()"
setBoolean ("build.printf_floating_point", useFFloatingPointBox.isSelected());
setBoolean ("build.scanf_floating_point", useSFloatingPointBox.isSelected());

// in showFrame()
useFFloatingPointBox.setSelected (getBoolean ("build.printf_floating_point"));
useSFloatingPointBox.setSelected (getBoolean ("build.scanf_floating_point"));



I know why the Arduino IDE developers haven't included such a useful option... it's SO complex!


Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 13, 2017, 02:12 am
@Jimmus
You actually need more than 43 precision is set to be more than zero! so you would need (precision + 43). precision is am unsigned 8 bit which can go all they way up to 256. also take note that copying from that array to the actual array would take some time too.

@krupski first comment (Nov 11, 2017, 07:41 am )
Well, the problem with heap is that it cannot be measured but I think it is unlikely for them to use heap.

@krupski (Nov 11, 2017, 08:04 am)
I tried but I failed as you can see in my other comment

@krupski  (Today at 12:06 am)
Have you written a guide on where to place these parts of your code so I can give it a shot.


Now on my progress in making a safe dtostrf:
1) I was not sure how should the function look like. So I start looking for function developed by others. soon I noticed dtostrf rarely is used and the alternative name usually used is ftoa. So, I looked up few ftoas and to all of them were safe. They may not have produce clean results like our dtostrf but they were safe.

2) I asked this question in AVR freaks and I was told to just implement it myself. So I started but problems were raised one after each other. The initial step was finding the files I need.
So here is a list:
ftoa_engine.h
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/ftoa_engine.h?revision=1218&view=markup (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/ftoa_engine.h?revision=1218&view=markup)

ftoa_engine.S
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/ftoa_engine.S?revision=2191&view=markup (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/ftoa_engine.S?revision=2191&view=markup)


macros.inc
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/macros.inc?revision=2542&view=markup (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/macros.inc?revision=2542&view=markup)

sectionname.h
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/sectionname.h?revision=2166&view=markup (http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/sectionname.h?revision=2166&view=markup)

#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.


So I started a new project and the code did not compile but not because something is missing. It could not understand the .S (assembly file) and the .INC file. So hen I remembered that Arduino IDE and assembly do not go well together.


I also thought of just using the assembly part in a function and use "asm volatile( // code );", so after finishing I noticed the code does not compile mainly because it cannot connect the function's arguments to the the variables in assembly code (even though I used the same name). Which kind of make sense.
So is there anyone here who knows how to connect the function arguments to assembly code? (we may need to relate the #defines to assembly code (I am not sure if this is done currently))
Or is there anyone who knows how we can trick the arduino IDE to use assembly code?
Or is there anyone who knows a way to use the "__ftoa_engine" function in Arduino IDE? Arduino dtosrf use this thing but the users cannot use it in the IDE environment.

My assembly function was too long, so I attached it.



Direction from here:
while I wait for someone who knows the solution to number 2, I start making an equivalent __ftoa_engine in C++ so that for the worst case scenario. Then I make the dtostrf which is safe. Any suggestions are appreciated.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 13, 2017, 02:21 am
Any suggestions are appreciated.
I'm watching this thread, my interest is just academic.

I have to say though, I'm trying to understand why it is you would need this.  I admit that I really couldn't understand from your explanation above.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 13, 2017, 02:45 am
Just to make things clearer after BulldogLowell saying that my explanation was not clear, I will try one more time to express what I meant. I think most uncertainty is in regard to part 2. So I try to explain in more detail.

In Arduino Freaks post (here (http://www.avrfreaks.net/forum/how-safely-and-reasonably-convert-float-or-double-string-or-char-array#comment-2320506)) They asked me to implement a dtostrf myself. I started doing it using the available resources fully. In current dtosrf most of the work is done by a function named "__ftoa_engine". This function is related to all the files in my list in the previous post. Unlike most libraries we so for Arduino, this function has its definition in a .h file and its implementation in a .S file. .S file is like a hybrid assembly + C/C++ file. So there are parts in C/C++ which are defines and things like that and then the implementation of the function is in assembly. The other 2 files also contain some stuff which are not you typical C++ file but I could live with that if everything would work out of the box(copy pasting code).
When I realized the files are being an issue, I thought of making one function with assembly code in it and trying to make it independent. So I made the file attached to my previous post having the whole ~500 lines of assembly using the "asm volatile( // code ); ". I read that this is the only way to use assembly in Arduino IDE. So I put my while code inside the code shown above (a lot of manual changes were needed to make the format compatible since this feature accept one line of assembly per each double quotation). After finishing the whole editing, I realized that the code does not compile because in assembly there are variable names which are the arguments passed to the function and these cannot be connected to the variables passed to the function. If you open my attached file you can see the following in the assembly code which do not get connected to the function's arguments: "val, buf, prec, maxdgs". Therefore I need someone to answer any of those 3 questions if I want to move forward with this approach.

I hope this clarified the process and the progress and everything else :D
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: BulldogLowell on Nov 13, 2017, 02:48 am
I hope this clarified the process and the progress and everything else :D
No, I understand your approach, I just don't understand (except for purely academic reasons) why you would need to do this.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 13, 2017, 03:32 am
@BulldogLowell I am having so much free time in my hand so let me explain one more time why there is a problem because I think you are not the only one not understanding it and also to be 100% I am not the one being wrong.

We call the current dtostrf. What happens is that this function does not know how big the buffer we want him to write in is. It could be any number but dtostrf does not have any clue about it. It could be larger, equal or smaller. If it is larger or equal, we are safe. If it is smaller then that is a bad news. The current dtostrf goes and write the number in the memory not knowing that the buffer is smaller. So the data in memory get corrupted since dtostrf does not stop writing in the memory when the small buffer has been used up.

One might say we know roughly how big our float is so we can make sure we have enough room. That is correct but the problem with float is that it can become very large if you miss few special cases. Let's say you expect your number to be between 0 to 10. Also let's assume you have set the precision (the way it works in current dtostrf) to 1. So the minimum will be "0.0" and the max will be "9.9". So, you want to be as efficient as you can get and you allocate an array with 3 bytes. (I am ignoring the null because dtostrf does not print null as far as I remember). You also set the width to 3.

if the number is 0 you will get     
if the number is 5.3 you will get  [5][.][3] (we are all good)
if the number is 9.9 you will get  [9][.][9] (we are all good)

Now if you variable for some reasons that you did not predict become something out of the range. let's see what happens:

if the number is -0.1 you will get   [-][0][.][1] (We changed some other data byte to '0')
if the number is 10.0 you will get  [1][0][.][0] (We changed some other data byte to '0')

or in some very ugly case:

if the number is 10000.0 you will get  [1][0][0][0][0][.][0] (We changed some other data bytes to '0' and one byte to '.')

So this is the problem. Now one solution is to put a huge buffer and the problem is that float variable can have a very huge value too. So, you won't be able to find a reasonable size. One way is to check and make sure if the number is in certain range. However the problem is that you should do this every time and on top of that you have to make sure you consider the sign and the precision (precision as it is in current dtostrf). So as you can see not all these solutions are neat.
If we had a dtostrf which would know the size of buffer, everything would be solved. dtostrf is technically aware of everything it is doing. It knows exactly how many bytes are being used and so one, so why not we just tell dtostrf how many byte it can use and then we sleep in peace at night not worrying about the float ending up out of range and screw the whole memory.

One more look at dtostrf


Quote
char* dtostrf   (   double    __val, signed char    __width, unsigned char    __prec, char *    __s)
Conversion is done in the format "[-]d.ddd". The minimum field width of the output string (including the possible '.' and the possible sign for negative values) is given in width, and prec determines the number of digits after the decimal sign. width is signed value, negative for left adjustment.
The dtostrf() function returns the pointer to the converted string s.
in the Arduino implemented:
__prec: determines the number of digits after the decimal point
What would make it safe although not as practical:
__prec: determines the number of digits after the sign


My suggestion for dtostrf:
Quote
bool dtostrf   (double __val, signed char __width, unsigned char __prec, char * __s, unsigned char __maxSize)
__val: same as current dtostrf
__width: same as current arduino dtostrf
__prec: same as current arduino dtostrf
__s: same as current arduino dtostrf
__maxSize: The number of bytes allocated for this number
return true if the process was successful and maxSize was large enough. Return false if the size was not enough.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 13, 2017, 05:53 am
Correction to my suggested dtostrf

Quote
char* dtostrf   (double __val, signed char __width, unsigned char __prec, char * __s, unsigned char __maxSize, bool __result)
__val: same as current dtostrf
__width: same as current arduino dtostrf
__prec: same as current arduino dtostrf
__s: same as current arduino dtostrf
__maxSize: The number of bytes allocated for this number
__result: true if the process was successful and maxSize was large enough. Return false if the size was not enough.
eturns the pointer to the converted string s. (this is used for sprintf and keep it compatible with what we have)
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 13, 2017, 09:53 am
Correction to my suggested dtostrf
eturns the pointer to the converted string s. (this is used for sprintf and keep it compatible with what we have)
You are making SUCH a big deal out of a VERY trivial thing.

And, any version of "dtostrf" does not need to be used with "sprintf". It already returns the formatted string (in a buffer of sufficient size that you have to supply - I hope you know).

As far as a "guide" being available to mod your IDE... all the information is in the post. I said which java source files are changed and what the changes are.

If I need to go into more detail than that (for example, how to re-compile the whole IDE) then I suggest not even trying.


Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 13, 2017, 10:04 am
Direction from here:
while I wait for someone who knows the solution to number 2, I start making an equivalent __ftoa_engine in C++ so that for the worst case scenario. Then I make the dtostrf which is safe. Any suggestions are appreciated.
Do you realize that what you are trying to do is duplicate the floating point support code that is already available in GCC?

This is too funny.... "GCC'c floating point uses up SO many resources so I think I'll just write my own".  (http://www.gunsnet.net/images/smilies/laughingtohard.gif)

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 14, 2017, 01:11 am
@krupski

I assume you have software background considering the conversation we had so far. In firmware this is not a forgiven small issue. I have worked in a place, where things smaller than this can make your manager go crazy. This is why it takes a lot more time to write firmware than software, you may go and read on this. It take significantly higher amount of time in average to produce a line in firmware than a line in software and the cost estimation for each line also very different mainly due to this.

I had once worked on a project where I was required to test a code for 72 hours straight with very critical timing. And oh boy a mistake like this would be the last thing you want. Imagine you test your code for 72 hours to figure out when you get to 71th hour the dtostrf does something crazy because of its range. It would probably so hard to find it.

And yeah I do not even know how to compile the code again. So, if you can point me to a guide that would be awesome so at the end I would do a comparison. I have never read a line of code in java nor touched a java compiler nor know how the whole thing works so there is not much I can do unless I spend significant amount of time to figure it out.

I do realize how sad it is that I cannot use the GCC (I think it is part of AVR-lib but it is not something that matters) floating point, but since no one had a solution I am making my own dtostrf.

And btw for the sprintf part, I am doing it for code compatibility and in case the sprintf was not modified to accept formatting.


My progress on dtostrf:

there are 3 ways to do this:
1) inefficient and in accurate:
This method is done by mathematically cracking up the float. A code example can be found here:
https://stackoverflow.com/questions/2302969/convert-a-float-to-a-string (https://stackoverflow.com/questions/2302969/convert-a-float-to-a-string)


2) More efficient with good accuracy (if not perfect) but limited range
In this method they use the largest integer supported on the software and then use ltoa to get the digits out. The limitation in range is due to 32bit or 64bit integer number. You can see the code here:
http://www.edaboard.com/thread5585.html (http://www.edaboard.com/thread5585.html)


3) the accurate and the most efficient version:
There are algorithm used to do dtostrf and apparently the back story is kind of interesting. The initial algorithm was Dragon 2 and then Dragon 4. However it was then improved further and was named Grisu. The Grisu 2 and Grisu 3 also exist based on some of the documents I have read. The best link regarding Grisu is here:
http://www.ryanjuckett.com/programming/printing-floating-point-numbers/ (http://www.ryanjuckett.com/programming/printing-floating-point-numbers/)


A bench mark comparison between these can be found here
https://github.com/miloyip/dtoa-benchmark (https://github.com/miloyip/dtoa-benchmark)

I either use method 3 or a hybrid version where less memory is used because Grisu needs some stuff stored in an array. At this point I am not sure what they are but I am reading on them. I also have a feeling that the arduino __floatEngine works with the same method because it appears to me that some numbers are stored however it could be method number 1. I can confirm this when I finish reading and understading Grisu
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: westfw on Nov 14, 2017, 05:52 am
I would be inclined to extract the existing code from the existing printf(), perhaps even using the existing standard formatting:
   ftostr(s, "%6.3f", x);

The arguments against printf() are mostly based on how much room is used by the functions/formats that you DONT use.  Extracting individual funcs solves that.

no one wants anyone to re-write long-proven complex asm code that was probably only written in asm in the first place because the C equiv was too big...
 
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 14, 2017, 06:23 am
@westfw

I have actually looked it up. Please correct me if I am wrong:

printf is basically a call to vfprintf and vfprintf has both assembly and __ftoa_engine usage.
Which means I go back to the same place where I started.

printf: https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#ga4c04da4953607fa5fa4d3908fecde449 (https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#ga4c04da4953607fa5fa4d3908fecde449)

vfprintf: https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#gaa3b98c0d17b35642c0f3e4649092b9f1 (https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#gaa3b98c0d17b35642c0f3e4649092b9f1)


I do not have too much experience with AVR assembly and considering the IDE support for assembly I gave up on that immediately.

I read more about Dragon and Grisu and I think it is a lot more complex than what we need in Arduino. They have actually introduced new data types of up to 35 uint32_t array (for 64 bit float). It is very precious but I do not think we can afford that much processing for dtostrf. The solution in second link is slightly better because it is based on itoa and therefore I have to do less work to come up with a new dtosrf and bigger part of it has been tested.

Honestly deep inside I wish I could use __ftoa_engine(), because then everything would be supper easy. sigh.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: westfw on Nov 14, 2017, 07:10 am
Quote
I do not have too much experience with AVR assembly and considering the IDE support for assembly I gave up on that immediately.
"I don't understand" is a poor excuse for giving up.



Quote
"__ftoa_engine" does most of the hard work and probably more efficient then what I could write. However I do not know how to include it or use it. If I learn to do that, the rest is done in few minutes. But I do not know how to include this file.
Figuring it out would be a better use of time than re-writing it.  As you say, adding a "max field width" to the ftoa_engine should be relatively trivial.


Quote
#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.
That's because this is Atmel-provided code rather than avr-libc, or something like that.
it shows up in the installed toolchain in .../avr/include/avr/io.h

Also, __ftoa_engine should certainly be in libc.  Make sure to declare it as a C function, in c++ or ino files:
Code: [Select]
extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 14, 2017, 09:36 am
@westfw

Quote
"I don't understand" is a poor excuse for giving up.
Well there is no way at least based on the replies I got over hear to write assembly code which takes variables from outside in Arduino IDE. let's say I have "uint8_t a;" how do I use this in my assembly code? This is why I gave up on it. If there is no way around that, nothing can be done as far I am concerned and in replies here and outside I could not find any solution for this problem.

Quote
Figuring it out would be a better use of time than re-writing it.  As you say, adding a "max field width" to the ftoa_engine should be relatively trivial
Actually  __ftoa_engine returns the exponent in decimal and the number digits as much as it is requested by the user (max) in a buffer. It is like scientific form but with exponent part being an integer. That is why I was hoping to find a way to use that in arduino IDE. But everything I tried failed. I even tried to copy paste the assembly code to get it working but due to the problem in previous paragraph it is not possible.

for more info on __ftoa_engine check:
https://sourcecodebrowser.com/avr-libc/1.8.0/ftoa__engine_8h.html#a0700887e129ad889bdff83eac78c8797 (https://sourcecodebrowser.com/avr-libc/1.8.0/ftoa__engine_8h.html#a0700887e129ad889bdff83eac78c8797)


Quote
#include <avr/io.h>
I faced other problems before getting to io.h as stated above.

"__ftoa_engine" is an internal function which IDE does not let me use it. It says it is not defined in this scope. I tried to add the related .h files but that also did not work. Unless I have made a mistake.

I searched for arduin __ftoa_engine and google has less than 20 pages showing up. 2 of it being this post.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: westfw on Nov 14, 2017, 10:13 am
Quote
"__ftoa_engine" is an internal function which IDE does not let me use it.
I just showed you how to use it...
Code: [Select]
extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);


Quote
there is no way over hear to write assembly code which takes variables from outside in Arduino IDE. let's say I have "uint8_t a;" how do I use this in my assembly code?
Of course there is.  This (along with dtostr() itself) is avr-gcc and C compiler stuff, and not "arduino stuff", so you have to expand your search.  Dealing with "uint8_t a" depends on exactly where it is.  If it's a function parameter, it will show up in one of the AVR registers (which one is based on its position in the parameter list.)  If it's global, it accessible via name.
See http://www.nongnu.org/avr-libc/user-manual/FAQ.html#faq_reg_usage (http://www.nongnu.org/avr-libc/user-manual/FAQ.html#faq_reg_usage)
Try this for a starter  (test.ino)

Code: [Select]
#include <stdio.h>
FILE* ser;
extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);

int fput(char c, FILE* f) {
  Serial.write( c);
  return 0;
}

void setup() {
  Serial.begin(9600);
  stdout = fdevopen(fput, NULL);
}

char buffer[20];
void ftoa_eng_study(float f)
{
  memset(buffer, 0, sizeof(buffer));
  int8_t exponent = __ftoa_engine(f, buffer, 8, 19);
  printf("Floating input according to Serial.print(): ");
  Serial.println(f, 8);
  printf("   engine Exponent: %d, flags 0x%02x Digits: ", exponent, buffer[0]);
  for (byte i = 1; i < 20; i++) {
    if (buffer[i] == 0) break;
    Serial.write(buffer[i]);
  }
  Serial.println();
  Serial.println();
}

void loop ()
{
  ftoa_eng_study(PI);
  ftoa_eng_study(987.654e20);
  ftoa_eng_study(42.0);
  ftoa_eng_study(6.626e-34);

  Serial.println();
  delay(10000);
}


(Code modified.   Hmm.  I don't know that I understand what happened to Plank's constant!)
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 14, 2017, 01:55 pm
@westfw Thank you so much, it is time for me to sleep, but I appreciate all the inputs. Sorry that I did not understand the fact that you are showing me how to use it.
Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: krupski on Nov 15, 2017, 10:54 pm
@westfw Thank you so much, it is time for me to sleep, but I appreciate all the inputs. Sorry that I did not understand the fact that you are showing me how to use it.
You have been shown many different ways of accomplishing your goal and you have simply written them off with lame statements such as "I don't understand it" or "it will take too many resources" while not even trying to actually do it.

We are all wasting our time with you and, as for me I'm done with this thread.

Title: Re: How to safely and reasonably convert a float or double to string or char array?
Post by: alirezasafdari on Nov 18, 2017, 03:42 am
@westfw thank you, I used the tip you gave me for __ftoa_engine. It was a new concept and I read about in here (http://www.geeksforgeeks.org/extern-c-in-c/ (http://www.geeksforgeeks.org/extern-c-in-c/)). Thank you again, without your guidance I would never learn that.

@krupski, sorry to let you down. I did not mean to. Anyways later if I release the libraries which I am working on right now, I think more people will be benefited if no tweaks are needed on the IDE side. Again I am sorry if I have made you upset.


Things I have done which might be useful for future users:
1) I documented the dtoa_prf function which is the actual function when dtostrf is called. The code does not have any documentation and I think it has been implemented in a very smart and efficient way and without documentation it is hard to understand it.
2) I made a safe version and tried to test it as much as I could in few aspects.
   i) dtosrf and myDtostrf produce the same exact results when size is not an issue. (it was tested for hours by generating random float numbers)
   ii) NAN, INF, -INF are tested with different sizes to make sure correct result is obtained
   iii) size was tested on some variables (I have not seen any suspicious behavior but I think more testing is required which hopefully happens over time when I am testing other parts of the project)
3) finding out that another person has faced the same issue but the whole thread is in german :)


My overall code is in here (well, it was too long so had to attach it)


myDtoa_prf part of this code is exactly same as dtoa_prf and buy reading the comments you can easily understand the original dtoa_prf too.

There is only one part which I did not understand and that is
Code: [Select]
if ((signed char)nDigits < 1)
nDigits = 1;
else if (nDigits > 8)
nDigits = 8;


I tested with
Code: [Select]
if ((signed char)nDigits < 0)
nDigits = 1;
else if (nDigits > 8)
nDigits = 8;


and it produce exact same results after testing with random float number, but I think it is faster because it can be branch if minus. (I am not very familiar with AVR assembly instruction yet but from what I recall from 6800 instruction set, branch if minus was faster than comparing and then branching)
I would be happy if anyone can point out why there is a 1 there instead of 0. I am afraid it is for a very special case and the random number generator has not generated that.

The code I have attached contain the testing section. Let me know if you think the testing part is not accurate, I think it is fairly accurate but just to be sure.


And lastly,
 https://www.mikrocontroller.net/topic/301125 (https://www.mikrocontroller.net/topic/301125)
 shows folks who solved the same problem. However my version has one difference. Their version does not print the number at all if it does not fit. My version it prints as much as it can and then set the bool argument which passed by reference to false.



I would like to thank you all for the support, suggestions and your efforts.