Go Down

Topic: String concatenation for Serial.print (Read 8128 times) previous topic - next topic

samtal

Hi,

Arduino Mega1280, Arduino standard IDE, include string lib.

To efficiently print out mixed data, I convert all data items into one concatenated string and send the string to the serial port.
This works nicely, except that I can not control the output format for an individual item as I could do in single item Serial.print(value, HEX).
Please see attached example.

Is there a way to format printout of individual concatenated items in a string?
Thx
 

krupski

Hi,

Arduino Mega1280, Arduino standard IDE, include string lib.

To efficiently print out mixed data, I convert all data items into one concatenated string and send the string to the serial port.
This works nicely, except that I can not control the output format for an individual item as I could do in single item Serial.print(value, HEX).
Please see attached example.

Is there a way to format printout of individual concatenated items in a string?
Thx
 
Use a buffer and the  sprintf  function.

Example

Code: [Select]

char buf [64]; // must be large enough for the whole string
int voltage; // variable to get a value from a voltage source (example)

// this is the code that generates the string to print
void loop (void) {
    sprintf (buf, "The voltage is currently %3d volts DC\r\n", voltage);
    Serial.print (buf);
    delay (1000); // print out voltage once a second
}


Assuming that the voltage is changing, you will see something like this in your serial monitor:

The voltage is currently  15 volts DC
The voltage is currently  14 volts DC
The voltage is currently  13 volts DC
The voltage is currently  12 volts DC
The voltage is currently  13 volts DC
The voltage is currently  14 volts DC
The voltage is currently  15 volts DC

See? The variable "voltage" is "sent to" the "%3d" format string and replaces it (the "3" means "make room for 3 digits" and the "d" means "the variable is an integer".

The %3 does this:
  0
  1
....
  9
 10
 11
...
 99
100
101

See how they line up?

Hope this helps.
Gentlemen may prefer Blondes, but Real Men prefer Redheads!

sterretje

To efficiently print out mixed data, I convert all data items into one concatenated string and send the string to the serial port.
Please define 'efficient'.
If you understand an example, use it.
If you don't understand an example, don't use it.

Electronics engineer by trade, software engineer by profession. Trying to get back into electronics after 15 years absence.

UKHeliBob

Please define 'efficient'.

My thoughts entirely.

Put the print statements in a suitably function and call it when required so that the main program looks neat and tidy and screen space is not "wasted" with a series of print statements.
Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

samtal

With 'efficient' I mean efficient in the serial port.
Instead of sending each line with a Serial.print command that has significant overhead, concatenating all data into one string and one print command cuts much of that overhead.

As to the last comment by Brattain Member, I did not relate to efficiency in the editor text.
The program lines are in one printout function. Of course, the lines in my example could be written as one line, but I like it my way, for clarity.

To the main point: My question was related to large number mixed values concatenating and formatting. Saving into a buffer can be an option, but not a nice one, and I need to test it to make sure I can format each individual value differently.


Delta_G

Your combining them has more overhead than just printing them one after another.  All print has to do is dump the data into a buffer.  Where it is concatenated like you want.  So you're basically making it do that twice.  Now which is really more efficient?
|| | ||| | || | ||  ~Woodstock

Please do not PM with technical questions or comments.  Keep Arduino stuff out on the boards where it belongs.

Delta_G

Quote
Saving into a buffer can be an option, but not a nice one
If you're not keeping it in some buffer then where are you concatenating it?  I think you have a general misunderstanding of what's going on here.  Perhaps you should describe what you actually want to see happen and let someone tell you what's the best and most efficient method to do it. 
|| | ||| | || | ||  ~Woodstock

Please do not PM with technical questions or comments.  Keep Arduino stuff out on the boards where it belongs.

-dev

Quote
Please define 'efficient'.
Indeed.

There is no reason to build one giant string and then print it all out at once.  There is no overhead associated with Serial.print that you can avoid by building one giant string.  Printing each piece by itself is actually much more efficient:

*  The Arduino can be printing the first part of the message while it formats the next parts.

*  No additional RAM is needed to contain any part of the printed string.  Only your variables will use RAM.

*  The characters to be printed are not copied into and out of this extra buffer; they are given directly to the Serial port object.

*  This avoids the additional processing required by String operations *and* avoids the numerous issues with String usage, especially long-term stability (read this).

Using sprintf also has disadvantages:

*  You must be very careful that the destination char buffer has enough room.  If you don't count right, the sprintf function will write past the end of the buffer, corrupting memory.  I don't know why people don't recommend snprintf to avoid this common problem.

*  The sprintf function is really a mini "interpreter".  It interprets the format string at run time and "executes" the various % formatting functions.  This is much slower than calling

    Serial.print( v, HEX )

*  Since the interpretation occurs at run-time, there is no compiler warning about trying to print an integer when you pass in a character (or any other mismatch between the % formatter and the argument you pass).

*  This format interpreter code is fairly long, adding about 1000 bytes to the executable size.  Because the format string can contain many different types, code for all types must be included in the executable.  When you use the print functions for individual pieces, the linker eliminates all the functions for types that you don't need.

*  Floating-point number are not supported by default.

For comparison, here are 3 short sketches that use each technique:

Code: (giant String) [Select]
void setup()
{
  Serial.begin( 9600 );
}

String string;
volatile int value;  // a trick to make sure the optimizer doesn't cheat  :)

// this is the code that generates the string to print
void loop (void) {
    value = millis() & 0xFF;

    unsigned long start = micros();

    string = "The value is currently 0x";
    string += hexDigit( value >> 4 );
    string += hexDigit( value );
    string += " units\r\n";

    Serial.print( string );

    Serial.print( micros() - start );
    Serial.println( F("us") );

    delay (1000); // print out value once a second
}

char hexDigit( int v )
{
  v &= 0x0F; // just the lower 4 bits
  if (v < 10)
    return '0' + v;
  else
    return 'A' + (v-10);
}

Code: (sprintf into buffer) [Select]
void setup()
{
  Serial.begin( 9600 );
}

char buf [64]; // must be large enough for the whole string
volatile int value;

// this is the code that generates the string to print
void loop (void) {
    value = millis() & 0xFF;

    unsigned long start = micros();

    sprintf (buf, "The value is currently 0x%02X units\r\n", value);
    Serial.print (buf);

    Serial.print( micros() - start );
    Serial.println( F("us") );

    delay (1000); // print out value once a second
}

Code: (piece-wise Serial.print) [Select]
volatile int value;

char hexDigit( int v )
{
  v &= 0x0F; // just the lower 4 bits
  if (v < 10)
    return '0' + v;
  else
    return 'A' + (v-10);
}

void printValue( int v )
{
    Serial.print( F("The value is currently 0x") );
    Serial.write( hexDigit( value >> 4 ) );
    Serial.write( hexDigit( value ) );
    Serial.println( F(" units") );
}

void setup()
{
  Serial.begin( 9600 );
}

void loop () {
    value = millis() & 0xFF;

    unsigned long start = micros();

    printValue( value );

    Serial.print( micros() - start );
    Serial.println( F("us") );

    delay (1000); // print out voltage once a second
}

The String version takes 370us to execute and uses 4384 bytes of program space and ~320 bytes of RAM (270 + ~50 bytes on the heap).
The sprintf version takes 560us to execute and uses 3868 bytes of program space and 300 bytes of RAM.
The piece-wise version takes 320us to execute and uses 2498 bytes of program space and 202 bytes of RAM.

Your choice, of course.  ;)

Cheers,
/dev
Really, I used to be /dev.  :(

UKHeliBob

Quote
As to the last comment by Brattain Member, I did not relate to efficiency in the editor text.
The program lines are in one printout function. Of course, the lines in my example could be written as one line, but I like it my way, for clarity.
If you are referring to my post then I am not suggesting that you put all of the print statements in one line, but I would certainly avoid using Strings as you do as it would lead to memory fragmentation.

I am not convinced that there is any significant overhead in using Serial.print more than once.  The baud rate of the Serial interface determines how fast each byte is sent and each byte is sent individually whether the data is all in one buffer, such as would be the case were sprintf() were used, whether it all in one String (or string) or whether several individual prints are done each with a different part of the data.
Please do not send me PMs asking for help.  Post in the forum then everyone will benefit from seeing the questions and answers.

sterretje

#9
Dec 12, 2017, 04:35 pm Last Edit: Dec 12, 2017, 04:36 pm by sterretje
Adding to all that

At the moment that the software output buffer for the serial port is full, your code will stall / block. That will be the case when using several print statements as well as in your approach.

If you use Serial.availableForWrite() and check if there is space to send some characters, you can prevent that from happening. Send a small chunk, check if there is enough space for the next chunk and send that etc. If there is not enough space, do something else.

With your approach, you can't.
If you understand an example, use it.
If you don't understand an example, don't use it.

Electronics engineer by trade, software engineer by profession. Trying to get back into electronics after 15 years absence.

krupski

To the main point: My question was related to large number mixed values concatenating and formatting. Saving into a buffer can be an option, but not a nice one, and I need to test it to make sure I can format each individual value differently.
Why is using a buffer not a "nice option"?  No matter how you print something, a temporary buffer is used,  Note that if you declare a buffer in a function, it's ram usage exists only as long as the function runs. The memory is freed when the function is complete.

What we all should be using is C's standard  "printf"  , but in keeping with the Arduino policy of not supporting essential functions in order to save half a dozen bytes, we are forced to either use a bunch of  "Serial.print (this)"  and  "Serial.print (that)"  functions, ad-nauseaum,  in order to print a simple, single line of text on the terminal or  sprintf  and a buffer.

And, because of the fear of using a few more bytes of memory by using standard in, standard out and standard error, burned into everyone's mind by people who don't have a clue what they are talking about, Arduino users won't even use a simple library that automatically provides stdin/out/err access claiming every reason from "uses a few more bytes" to "it blocks" to "it will stop the sun from shining" to "the IDE doesn't support it" (when, of course, the IDE doesn't support ANYTHING... AVR-GCC does and indeed AVR-GCC does support all C/C++ functions) and instead happily go on typing ridiculous stuff like this:

Code: [Select]

int volts = 120;

Serial.print ("Voltage: ");

if (volts < 10) {
    Serial.print (" "); // align columns
}

if (volts < 100) {
    Serial.print (" "); // align 100's place
}

Serial.print (volts);

Serial.print (" VDC");

Serial.print ('\r'); // goto...
Serial.print ('\n'); // ...new line because the Print library doesn't even
    // know how to translate a Unix newline into a CR/LF.

 
When they COULD just do this:

Code: [Select]

int volts = 120;
fprintf (stdout, "Voltage: %3d VDC\n", volts);


Don't know why... maybe there's some perverse pleasure in repeatedly ramming one's head into the wall.... ?

And, if the user doesn't want to install a simple library to make things 1000% easier, the next best method is to use a temporary buffer and  "sprintf"  which is almost as good (but not AS good) as using printf directly.

While we're at it, the Arduino developers, in their wisdom(?) not only disable floating point print support without providing the option of using it if desired, they also espouse the use of "dtostrf" which is complicated, not understood by a lot of people, doesn't fully support "printf style" formatting and requires the user to provide a temporary buffer for it (and the user is responsible for making sure the buffer is large enough).

Everyone shies away from floating point because it uses a few dozen more bytes of memory, but the fact that "dtostrf" also consumes program and ram space doesn't seem to bother anyone.

I feel sorry for any Arduino user who ends up programming for a living, because they will forever be hampered by all the convoluted or just plain wrong "facts" they learned from the "experts".  Learning something new is tough enough without having to also UNLEARN the wrong stuff the "experts" taught them.

...and don't even get me started on the absurdity of worrying about a function "blocking" when the code is single tasking/ single threaded and running on a toy microcontroller with a quarter meg or less of memory (or, nuttiness in the other direction...) setting up a bunch of interrupt handlers to read the state of a switch or blink an LED on and off...  
Gentlemen may prefer Blondes, but Real Men prefer Redheads!

krupski

Indeed.

There is no reason to build one giant string and then print it all out at once.   [1] There is no overhead associated with Serial.print that you can avoid by building one giant string.  [2] Printing each piece by itself is actually much more efficient:

*  [3] The Arduino can be printing the first part of the message while it formats the next parts.
I know that "boldly asserted is half proven", but I would love to see some proof (links, whatever) to back up those 3 assertions (because you are wrong on all 3 counts).
Gentlemen may prefer Blondes, but Real Men prefer Redheads!

DrAzzy

I know that "boldly asserted is half proven", but I would love to see some proof (links, whatever) to back up those 3 assertions (because you are wrong on all 3 counts).
Well, he did show three example programs using the different techniques, and the version with the giant string, and the one with sprintf both used about 50% more ram, 100% more flash, and executed more slowly. What sort of proof are you looking for?
ATTinyCore for x4/x5/x61/x7/x8/x41/1634/828/x313 megaTinyCore for the megaavr ATtinies - Board Manager:
http://drazzy.com/package_drazzy.com_index.json
ATtiny breakouts, mosfets, awesome prototyping board in my store http://tindie.com/stores/DrAzzy

Delta_G

#13
Dec 13, 2017, 11:50 pm Last Edit: Dec 13, 2017, 11:51 pm by Delta_G
I know that "boldly asserted is half proven", but I would love to see some proof (links, whatever) to back up those 3 assertions (because you are wrong on all 3 counts).
Asking for proof and claiming he's wrong... with no proof.  -dev showed the code and results.  What more proof do you want?  What proof have you other than your general distaste for the Arduino community?   Does your version with printf produce smaller or faster code?  Prove it. 
|| | ||| | || | ||  ~Woodstock

Please do not PM with technical questions or comments.  Keep Arduino stuff out on the boards where it belongs.

westfw

There are cases where pre-buffering print() output might be useful.  AFAIK, neither Ethernet.print() nor USBSerial.print() does anything intelligent in terms of avoiding the "one message per print statement" problem, and you could potentially improve throughput quite a lot.  Normal HardwareSerial.print() would gain very little, though; in additional to there being no smarts for "aggregating" small output requests, there also aren't any smarts for optimizing large requests.

Go Up