How to safely and reasonably convert a float or double to string or char array?

alirezasafdari:
Sorry krupski. I just stated what I have read and did not test it myself. I give it a shot now. Just one question, does the new files use heap? I am going to test it anyway but just wanted to know if they use heap.

They are not "new" files, you are simply tricking GCC into linking in the floating point code when it's actually trying to link in the "minimal" (non-floating point) code.

As far as "heap", I don't really know or care how GCC allocates memory. I only care that I have ENOUGH.

That's why I suggested to build the test program first, then see how much flash and ram it uses, then do the file rename trick to use the floating point code and build the exact same test program again and compare flash and ram usage.

If you like how it works and you feel the little extra memory usage is OK, then stick with it. If not, simply rename the files back to original and all is back to the way it was before.

westfw:
It's worth remembering why printf() and floatingPoint printf() are not the defaults. a printf() based implementation of "hello world" is about 1k bigger than the same with Serial.print(), and using the floating point version of printf() expands the difference to about 4kbytes, WHETHER OR NOT YOU USE THE FLOATING POINT FEATURES. Those were big numbers back when a "large" AVR only had 4k to 8k of code space.

Did you see my [u]post[/u] a few places up?

There are TWO different floating point support files... one for printf and related, another for scanf and related.

There is no need to link in BOTH of them, You can use one, the other or both. And, since scanf and related are rarely used, one can link in ONLY the printf floating point code and save quite a bit of space.

That's why I have the option to chose one, the other or both in my Preferences.

(click image for full size)
prefs.jpg

@krupski

I have tried your trick by renaming the files and it did not work. I think it was simple enough for me not to make a mistake but again I could be wrong. (I am using latest Arduino).

However since I did a small test, I decided to share the result (not sure if it is meaningful but again it is a job done)

For Dtostrf:

#include "MemoryFree.h"

void setup() 
{
 int availableMemory;
 availableMemory = freeMemory();

 Serial.begin(115200);
 uint8_t buffer[60] = {};
 float inputNumber = 1357.125;
 uint32_t start, end;
 
 Serial.println(availableMemory);
 
 start = micros();
 dtostrf(inputNumber, 0, 3, buffer);
 end = micros();
 
 Serial.println(end - start);

 Serial.write(buffer, 60);
 
}
// the loop function runs over and over again forever
void loop() 
{

 
}

Result:

Using dtostrf:
Flash space = 3790
Global variable = 190
Free memory at the beginning = 7937
Micros taken = 120
Output is: 1357.125 [00] X 52

for sprintf:

#include "MemoryFree.h"

void setup() 
{
 int availableMemory;
 availableMemory = freeMemory();

 Serial.begin(115200);
 uint8_t buffer[60] = {};
 float inputNumber = 1357.125;
 uint32_t start, end;
 
 Serial.println(availableMemory);
 
 start = micros();
 sprintf (buffer, "%7.3f\r", inputNumber);
 end = micros();
 
 Serial.println(end - start);

 Serial.write(buffer, 60);
 
}

// the loop function runs over and over again forever
void loop() 
{

 
}

results:

Without modifying the IDE:
Flash space = 3766
Global variable = 196
Free memory at the beginning = 7931
Micros taken = 80
Output is: [20] [20] [20] [20] [20] [20][3F = ‘?’][0D = ‘\n’] [00] X 52

With modifying the IDE:
Flash space = 3766
Global variable = 196
Free memory at the beginning = 7931
Micros taken = 80
Output is: [20] [20] [20] [20] [20] [20][3F = ‘?’][0D = ‘\n’] [00] X 52

@westfw
I am not sure about (avr-libc documentation) but the the other stostrf I found they let you set the number of digits after the sign. This technically solve all the problems. I am not sure what they do if they cannot fit it in the given number of digits but they definitely have a way to know.

__ftoa_engine() precision is different from in current arduino dtostrf. the precision tells the maximum number of digit coming out. that is why they put a 7 there to indicate the want only 7 numbers to be printed in a 9 byte buffer. Not sure about the return though. The implementation is odd with zero comments which make it tougher.

krupski:
Rename the file "[b]libprintf_min.a[/b]" to "[b]libprintf_min.a.backup[/b]".
Copy the file "[b]libprintf_flt.a[/b]" to "[b]libprintf_min.a[/b]"

libprintf_min.a is not the default printf, but an even more stripped down version.
The default version is in libc.a.

oqibidipo:
libprintf_min.a is not the default printf, but an even more stripped down version.
The default version is in libc.a.

Here is the exact piece of code that I have in "Compiler.java" to do the job:

///////////////// this provides the floating point option /////////////////

if (Preferences.getBoolean ("build.printf_floating_point")) {
    baseCommandLinker.add ("-Wl,-u,vfprintf,-lprintf_flt");
}

if (Preferences.getBoolean ("build.scanf_floating_point")) {
    baseCommandLinker.add ("-Wl,-u,vfscanf,-lscanf_flt");
}

///////////////// this provides the floating point option /////////////////

...and this code provides the check boxes in "Preferences,java" to select or deselect either one:

// [ ] printf Floating point support
useFFloatingPointBox = new JCheckBox (("  Enable (f)printf floating point"));
useFFloatingPointBox.setToolTipText (getToolTip ("Enable floating point support for printf"));
pane.add (useFFloatingPointBox);
d = useFFloatingPointBox.getPreferredSize();
useFFloatingPointBox.setBounds (left, top, d.width + 10, d.height);
right = Math.max (right, left + d.width);
top += d.height + GUI_BETWEEN;

// [ ] scanf Floating point support
useSFloatingPointBox = new JCheckBox (("  Enable (f)scanf floating point"));
useSFloatingPointBox.setToolTipText (getToolTip ("Enable floating point support for scanf"));
pane.add (useSFloatingPointBox);
d = useSFloatingPointBox.getPreferredSize();
useSFloatingPointBox.setBounds (left, top, d.width + 10, d.height);
right = Math.max (right, left + d.width);
top += d.height + GUI_BETWEEN;

And finally this code reads and writes the preferences in "Preferences.txt":

// in "applyFrame()"
setBoolean ("build.printf_floating_point", useFFloatingPointBox.isSelected());
setBoolean ("build.scanf_floating_point", useSFloatingPointBox.isSelected());

// in showFrame()
useFFloatingPointBox.setSelected (getBoolean ("build.printf_floating_point"));
useSFloatingPointBox.setSelected (getBoolean ("build.scanf_floating_point"));

I know why the Arduino IDE developers haven't included such a useful option... it's SO complex!

@Jimmus
You actually need more than 43 precision is set to be more than zero! so you would need (precision + 43). precision is am unsigned 8 bit which can go all they way up to 256. also take note that copying from that array to the actual array would take some time too.

@krupski first comment (Nov 11, 2017, 07:41 am )
Well, the problem with heap is that it cannot be measured but I think it is unlikely for them to use heap.

@krupski (Nov 11, 2017, 08:04 am)
I tried but I failed as you can see in my other comment

@krupski (Today at 12:06 am)
Have you written a guide on where to place these parts of your code so I can give it a shot.

Now on my progress in making a safe dtostrf:

  1. I was not sure how should the function look like. So I start looking for function developed by others. soon I noticed dtostrf rarely is used and the alternative name usually used is ftoa. So, I looked up few ftoas and to all of them were safe. They may not have produce clean results like our dtostrf but they were safe.

  2. I asked this question in AVR freaks and I was told to just implement it myself. So I started but problems were raised one after each other. The initial step was finding the files I need.
    So here is a list:
    ftoa_engine.h
    [avr-libc] Contents of /trunk/avr-libc/common/ftoa_engine.h

ftoa_engine.S
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/libc/stdlib/ftoa_engine.S?revision=2191&view=markup

macros.inc
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/macros.inc?revision=2542&view=markup

sectionname.h
http://svn.savannah.gnu.org/viewvc/avr-libc/trunk/avr-libc/common/sectionname.h?revision=2166&view=markup

#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.

So I started a new project and the code did not compile but not because something is missing. It could not understand the .S (assembly file) and the .INC file. So hen I remembered that Arduino IDE and assembly do not go well together.

I also thought of just using the assembly part in a function and use "asm volatile( // code );", so after finishing I noticed the code does not compile mainly because it cannot connect the function's arguments to the the variables in assembly code (even though I used the same name). Which kind of make sense.
So is there anyone here who knows how to connect the function arguments to assembly code? (we may need to relate the #defines to assembly code (I am not sure if this is done currently))
Or is there anyone who knows how we can trick the arduino IDE to use assembly code?
Or is there anyone who knows a way to use the "__ftoa_engine" function in Arduino IDE? Arduino dtosrf use this thing but the users cannot use it in the IDE environment.

My assembly function was too long, so I attached it.

Direction from here:
while I wait for someone who knows the solution to number 2, I start making an equivalent __ftoa_engine in C++ so that for the worst case scenario. Then I make the dtostrf which is safe. Any suggestions are appreciated.

floatEngineTest.ino (11.2 KB)

alirezasafdari:
Any suggestions are appreciated.

I'm watching this thread, my interest is just academic.

I have to say though, I'm trying to understand why it is you would need this. I admit that I really couldn't understand from your explanation above.

Just to make things clearer after BulldogLowell saying that my explanation was not clear, I will try one more time to express what I meant. I think most uncertainty is in regard to part 2. So I try to explain in more detail.

In Arduino Freaks post (here) They asked me to implement a dtostrf myself. I started doing it using the available resources fully. In current dtosrf most of the work is done by a function named "__ftoa_engine". This function is related to all the files in my list in the previous post. Unlike most libraries we so for Arduino, this function has its definition in a .h file and its implementation in a .S file. .S file is like a hybrid assembly + C/C++ file. So there are parts in C/C++ which are defines and things like that and then the implementation of the function is in assembly. The other 2 files also contain some stuff which are not you typical C++ file but I could live with that if everything would work out of the box(copy pasting code).
When I realized the files are being an issue, I thought of making one function with assembly code in it and trying to make it independent. So I made the file attached to my previous post having the whole ~500 lines of assembly using the "asm volatile( // code ); ". I read that this is the only way to use assembly in Arduino IDE. So I put my while code inside the code shown above (a lot of manual changes were needed to make the format compatible since this feature accept one line of assembly per each double quotation). After finishing the whole editing, I realized that the code does not compile because in assembly there are variable names which are the arguments passed to the function and these cannot be connected to the variables passed to the function. If you open my attached file you can see the following in the assembly code which do not get connected to the function's arguments: "val, buf, prec, maxdgs". Therefore I need someone to answer any of those 3 questions if I want to move forward with this approach.

I hope this clarified the process and the progress and everything else :smiley:

alirezasafdari:
I hope this clarified the process and the progress and everything else :smiley:

No, I understand your approach, I just don't understand (except for purely academic reasons) why you would need to do this.

@BulldogLowell I am having so much free time in my hand so let me explain one more time why there is a problem because I think you are not the only one not understanding it and also to be 100% I am not the one being wrong.

We call the current dtostrf. What happens is that this function does not know how big the buffer we want him to write in is. It could be any number but dtostrf does not have any clue about it. It could be larger, equal or smaller. If it is larger or equal, we are safe. If it is smaller then that is a bad news. The current dtostrf goes and write the number in the memory not knowing that the buffer is smaller. So the data in memory get corrupted since dtostrf does not stop writing in the memory when the small buffer has been used up.

One might say we know roughly how big our float is so we can make sure we have enough room. That is correct but the problem with float is that it can become very large if you miss few special cases. Let's say you expect your number to be between 0 to 10. Also let's assume you have set the precision (the way it works in current dtostrf) to 1. So the minimum will be "0.0" and the max will be "9.9". So, you want to be as efficient as you can get and you allocate an array with 3 bytes. (I am ignoring the null because dtostrf does not print null as far as I remember). You also set the width to 3.

if the number is 0 you will get [0]- [0] (we are all good)
if the number is 5.3 you will get [5]- [3] (we are all good)
if the number is 9.9 you will get [9]- [9] (we are all good)

Now if you variable for some reasons that you did not predict become something out of the range. let's see what happens:

if the number is -0.1 you will get [-][0]- [1] (We changed some other data byte to '0')
if the number is 10.0 you will get [1][0]- [0] (We changed some other data byte to '0')

or in some very ugly case:

if the number is 10000.0 you will get [1][0][0][0][0]- [0] (We changed some other data bytes to '0' and one byte to '.')

So this is the problem. Now one solution is to put a huge buffer and the problem is that float variable can have a very huge value too. So, you won't be able to find a reasonable size. One way is to check and make sure if the number is in certain range. However the problem is that you should do this every time and on top of that you have to make sure you consider the sign and the precision (precision as it is in current dtostrf). So as you can see not all these solutions are neat.
If we had a dtostrf which would know the size of buffer, everything would be solved. dtostrf is technically aware of everything it is doing. It knows exactly how many bytes are being used and so one, so why not we just tell dtostrf how many byte it can use and then we sleep in peace at night not worrying about the float ending up out of range and screw the whole memory.

One more look at dtostrf

char* dtostrf ( double __val, signed char __width, unsigned char __prec, char * __s)
Conversion is done in the format "[-]d.ddd". The minimum field width of the output string (including the possible '.' and the possible sign for negative values) is given in width, and prec determines the number of digits after the decimal sign. width is signed value, negative for left adjustment.
The dtostrf() function returns the pointer to the converted string s.

in the Arduino implemented:
__prec: determines the number of digits after the decimal point
What would make it safe although not as practical:
__prec: determines the number of digits after the sign

My suggestion for dtostrf:

bool dtostrf (double __val, signed char __width, unsigned char __prec, char * __s, unsigned char __maxSize)
__val: same as current dtostrf
__width: same as current arduino dtostrf
__prec: same as current arduino dtostrf
__s: same as current arduino dtostrf
__maxSize: The number of bytes allocated for this number
return true if the process was successful and maxSize was large enough. Return false if the size was not enough.

Correction to my suggested dtostrf

char* dtostrf (double __val, signed char __width, unsigned char __prec, char * __s, unsigned char __maxSize, bool __result)
__val: same as current dtostrf
__width: same as current arduino dtostrf
__prec: same as current arduino dtostrf
__s: same as current arduino dtostrf
__maxSize: The number of bytes allocated for this number
__result: true if the process was successful and maxSize was large enough. Return false if the size was not enough.

eturns the pointer to the converted string s. (this is used for sprintf and keep it compatible with what we have)

alirezasafdari:
Correction to my suggested dtostrf
eturns the pointer to the converted string s. (this is used for sprintf and keep it compatible with what we have)

You are making SUCH a big deal out of a VERY trivial thing.

And, any version of "dtostrf" does not need to be used with "sprintf". It already returns the formatted string (in a buffer of sufficient size that you have to supply - I hope you know).

As far as a "guide" being available to mod your IDE... all the information is in the post. I said which java source files are changed and what the changes are.

If I need to go into more detail than that (for example, how to re-compile the whole IDE) then I suggest not even trying.

alirezasafdari:
Direction from here:
while I wait for someone who knows the solution to number 2, I start making an equivalent __ftoa_engine in C++ so that for the worst case scenario. Then I make the dtostrf which is safe. Any suggestions are appreciated.

Do you realize that what you are trying to do is duplicate the floating point support code that is already available in GCC?

This is too funny.... "GCC'c floating point uses up SO many resources so I think I'll just write my own".

@krupski

I assume you have software background considering the conversation we had so far. In firmware this is not a forgiven small issue. I have worked in a place, where things smaller than this can make your manager go crazy. This is why it takes a lot more time to write firmware than software, you may go and read on this. It take significantly higher amount of time in average to produce a line in firmware than a line in software and the cost estimation for each line also very different mainly due to this.

I had once worked on a project where I was required to test a code for 72 hours straight with very critical timing. And oh boy a mistake like this would be the last thing you want. Imagine you test your code for 72 hours to figure out when you get to 71th hour the dtostrf does something crazy because of its range. It would probably so hard to find it.

And yeah I do not even know how to compile the code again. So, if you can point me to a guide that would be awesome so at the end I would do a comparison. I have never read a line of code in java nor touched a java compiler nor know how the whole thing works so there is not much I can do unless I spend significant amount of time to figure it out.

I do realize how sad it is that I cannot use the GCC (I think it is part of AVR-lib but it is not something that matters) floating point, but since no one had a solution I am making my own dtostrf.

And btw for the sprintf part, I am doing it for code compatibility and in case the sprintf was not modified to accept formatting.

My progress on dtostrf:

there are 3 ways to do this:

  1. inefficient and in accurate:
    This method is done by mathematically cracking up the float. A code example can be found here:
    c++ - Convert a float to a string - Stack Overflow

  2. More efficient with good accuracy (if not perfect) but limited range
    In this method they use the largest integer supported on the software and then use ltoa to get the digits out. The limitation in range is due to 32bit or 64bit integer number. You can see the code here:
    ftoa() implementation | Forum for Electronics

  3. the accurate and the most efficient version:
    There are algorithm used to do dtostrf and apparently the back story is kind of interesting. The initial algorithm was Dragon 2 and then Dragon 4. However it was then improved further and was named Grisu. The Grisu 2 and Grisu 3 also exist based on some of the documents I have read. The best link regarding Grisu is here:
    Printing Floating-Point Numbers - RyanJuckett.com

A bench mark comparison between these can be found here

I either use method 3 or a hybrid version where less memory is used because Grisu needs some stuff stored in an array. At this point I am not sure what they are but I am reading on them. I also have a feeling that the arduino __floatEngine works with the same method because it appears to me that some numbers are stored however it could be method number 1. I can confirm this when I finish reading and understading Grisu

I would be inclined to extract the existing code from the existing printf(), perhaps even using the existing standard formatting:
ftostr(s, "%6.3f", x);

The arguments against printf() are mostly based on how much room is used by the functions/formats that you DONT use. Extracting individual funcs solves that.

no one wants anyone to re-write long-proven complex asm code that was probably only written in asm in the first place because the C equiv was too big...

@westfw

I have actually looked it up. Please correct me if I am wrong:

printf is basically a call to vfprintf and vfprintf has both assembly and __ftoa_engine usage.
Which means I go back to the same place where I started.

printf: https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#ga4c04da4953607fa5fa4d3908fecde449

vfprintf: https://sourcecodebrowser.com/avr-libc/1.8.0/group__avr__stdio.html#gaa3b98c0d17b35642c0f3e4649092b9f1

I do not have too much experience with AVR assembly and considering the IDE support for assembly I gave up on that immediately.

I read more about Dragon and Grisu and I think it is a lot more complex than what we need in Arduino. They have actually introduced new data types of up to 35 uint32_t array (for 64 bit float). It is very precious but I do not think we can afford that much processing for dtostrf. The solution in second link is slightly better because it is based on itoa and therefore I have to do less work to come up with a new dtosrf and bigger part of it has been tested.

Honestly deep inside I wish I could use __ftoa_engine(), because then everything would be supper easy. sigh.

I do not have too much experience with AVR assembly and considering the IDE support for assembly I gave up on that immediately.

"I don't understand" is a poor excuse for giving up.

"__ftoa_engine" does most of the hard work and probably more efficient then what I could write. However I do not know how to include it or use it. If I learn to do that, the rest is done in few minutes. But I do not know how to include this file.

Figuring it out would be a better use of time than re-writing it. As you say, adding a "max field width" to the ftoa_engine should be relatively trivial.

#include <avr/io.h>
I could not find this one though. I thought this is supposed to be the easiest one.

That's because this is Atmel-provided code rather than avr-libc, or something like that.
it shows up in the installed toolchain in .../avr/include/avr/io.h

Also, __ftoa_engine should certainly be in libc. Make sure to declare it as a C function, in c++ or ino files:

extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);

@westfw

"I don't understand" is a poor excuse for giving up.

Well there is no way at least based on the replies I got over hear to write assembly code which takes variables from outside in Arduino IDE. let's say I have "uint8_t a;" how do I use this in my assembly code? This is why I gave up on it. If there is no way around that, nothing can be done as far I am concerned and in replies here and outside I could not find any solution for this problem.

Figuring it out would be a better use of time than re-writing it. As you say, adding a "max field width" to the ftoa_engine should be relatively trivial

Actually __ftoa_engine returns the exponent in decimal and the number digits as much as it is requested by the user (max) in a buffer. It is like scientific form but with exponent part being an integer. That is why I was hoping to find a way to use that in arduino IDE. But everything I tried failed. I even tried to copy paste the assembly code to get it working but due to the problem in previous paragraph it is not possible.

for more info on __ftoa_engine check:
https://sourcecodebrowser.com/avr-libc/1.8.0/ftoa__engine_8h.html#a0700887e129ad889bdff83eac78c8797

#include <avr/io.h>

I faced other problems before getting to io.h as stated above.

"__ftoa_engine" is an internal function which IDE does not let me use it. It says it is not defined in this scope. I tried to add the related .h files but that also did not work. Unless I have made a mistake.

I searched for arduin __ftoa_engine and google has less than 20 pages showing up. 2 of it being this post.

"__ftoa_engine" is an internal function which IDE does not let me use it.

I just showed you how to use it...

extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);

there is no way over hear to write assembly code which takes variables from outside in Arduino IDE. let's say I have "uint8_t a;" how do I use this in my assembly code?

Of course there is. This (along with dtostr() itself) is avr-gcc and C compiler stuff, and not "arduino stuff", so you have to expand your search. Dealing with "uint8_t a" depends on exactly where it is. If it's a function parameter, it will show up in one of the AVR registers (which one is based on its position in the parameter list.) If it's global, it accessible via name.
See Frequently Asked Questions
Try this for a starter (test.ino)

#include <stdio.h>
FILE* ser;
extern "C" int __ftoa_engine (double val, char *buf, unsigned char prec, unsigned char maxdgs);

int fput(char c, FILE* f) {
  Serial.write( c);
  return 0;
}

void setup() {
  Serial.begin(9600);
  stdout = fdevopen(fput, NULL);
}

char buffer[20];
void ftoa_eng_study(float f)
{
  memset(buffer, 0, sizeof(buffer));
  int8_t exponent = __ftoa_engine(f, buffer, 8, 19);
  printf("Floating input according to Serial.print(): ");
  Serial.println(f, 8);
  printf("   engine Exponent: %d, flags 0x%02x Digits: ", exponent, buffer[0]);
  for (byte i = 1; i < 20; i++) {
    if (buffer[i] == 0) break;
    Serial.write(buffer[i]);
  }
  Serial.println();
  Serial.println();
}

void loop ()
{
  ftoa_eng_study(PI);
  ftoa_eng_study(987.654e20);
  ftoa_eng_study(42.0);
  ftoa_eng_study(6.626e-34);

  Serial.println();
  delay(10000);
}

(Code modified. Hmm. I don't know that I understand what happened to Plank's constant!)

@westfw Thank you so much, it is time for me to sleep, but I appreciate all the inputs. Sorry that I did not understand the fact that you are showing me how to use it.