In my Linux world, that '\r' after the '\n' in the Arduino Serial.println() command gives an extra line I don't want. Since my sketch becomes longer for each Serial.print() command compiled, and I need to minimize the size of my large sketch, I would prefer to modify the behavior of the Serial.println() command rather than send the ( char )10 taking up another call to Serial.print(). Has anyone successfully found a way to prevent the println() command from sending the '\r'? Like monitoring the Serial output buffer with some ISR piggyback code and clobbering all '\r' characters found before they are sent out? Right now I'm using scores of this line to do what I want:
Dig through the hardware serial library for starters and follow the included libraries if needed. Not sure where the println is defined (might be print.h) and modify the println method.
Note that if this is in the print library, it will affect all other libraries that make use of that.
Because Serial.print can take a wide variety of types as arguments, and sometimes more than one argument, I used a template parameter pack to handle that. Because some Arduinos have more than one Serial, the first argument is a reference to exactly which one you want to print to. All arguments after that are passed directly to Serial. Because it's a template function, it should work with all of the print overloads without having to explicitly declare a function for each type. Even the F() macro should work just fine as is.
Example sketch:
template<class... Types> void linuxPrintLn( HardwareSerial &s, Types ... args )
{
s.print( args... );
s.write( '\r' );
}
void setup() {
// put your setup code here, to run once:
Serial.begin(250000);
linuxPrintLn(Serial, 1000, HEX);
}
void loop() {
// put your main code here, to run repeatedly:
}
Jiggy-Ninja and sterretje's are following the same line of thought and are a excellent answers, more along the lines of what I was thinking I'd need to do. sterretje, your answer is less portable than what I'm hoping for, so I don't consider it to be my first-choice go-to.
westfw, I question the ease of portability of your first idea. I do use the stty to Tx to the Arduino and the Linux version does allow the igncr flag BUT I'm not using stty for Rx - I'm using cat. I don't even know if I can run stty nohup and & (background daemon) as easily as the cat does it, and I'm taking your word for it that it works as deamon for Rx even while a separate process uses another instance of it to Tx. My cat process has been working reliably for a year or two, so I'm a little less inclined to experiment with changing to stty. Very excellent solutions, however, and the stty flag could end up being my permanent solution.
TonyWilk, good work, but I question whether compiling your solution gets me a smaller sketch footprint. I CERTAINLY would try it, however, in the absence of the other two answers.
Robin2, you can see the behavior when receiving Arduino Serial at a Linux box using cat. I noticed way back when that there was a difference between two of my Arduinos in the way they line-ended their Serial out, so maybe you have one that defaults to not having that '\n'? Anyway, I've mostly forgotten the details of how I came to notice it. My two Arduinos may even have had different sketches from each other in them, but I somehow discovered that println() was giving me a blank line every time from at least one of the boards. I could easily have missed the subtlety if I hadn't had the two different boards, but I'm not certain now that the difference was hardware only.
I'm planning on implementing Jiggy-Ninja's idea first and check compiled sketch size. Could be tomorrow before I get around to it. Thank you all for being amazingly helpful!
Jiggy-Ninja:
Actually, Linux uses LF-only as it's EOL convention, so the \r in my example needs to be changed to \n.
Copy that good buddy
Robin2:
I've never had any need for that.
...R
Robin2, do you use stty instead? If so, do 2 instances of stty play well together on the same tty? I use CLI on the box I speak of, so no IDE possible on the production box. Thanks!
kenneth558,
You must be in some strange world.
As a few others have said, \n IS the proper/normal line ending for *nix machines.
Unix/Linux/*nix systems have used \n as the standard line ending character since the beginning, i.e. for close to 50 years.
The use of carriage returns for line endings comes from other operating systems like CPM which DOS adopted and is used by Windows, and MAC.
DOS uses both, and MAC uses just \r
They f'd up.
Now if you are complaining about the extra unneeded \r character, that is another matter.
However, that can also easily be dealt in multiple ways with since you are using linux.
Jut my opinion, but I think you need to rethink your approach.
There are already tools and ways to handle this stuff like using stty to set line disaplines in the serial driver and you should try to use those rather than re-inventing a new solution.
But you could also pipe the output of cat to dos2unix to strip out the carriage returns before you use the character stream and dos2unix would probably work instead of using cat.
Or you could simply write a tiny program on the linux side that opens the port (just like cat) but also does what needs to be done to set the line disaplines to ignore carriage returns.
*unix systems have lots of flexibility in ways to handle things.
If you are still dead set on modifying the Arduino side, then just take Westfw's advice and go modify the Print class println() function.
Change it to only send \n by deleting the line of code that sends the \r.
IMO, I'd fix things on the linux side as it is much simpler and doesn't require any hacks on the Arduino side.
bperrybap:
IMO, I'd fix things on the linux side as it is much simpler and doesn't require any hacks on the Arduino side.
Agree totally, just hadn't come across a way in my particular situation. So you're saying that the output from cat retains the incoming line's line-end characters? I didn't realize that - guess I wasn't thinking clearly enough to even question it. Understand that I don't want to categorically remove all blank lines from the cat STDOUT (which goes from there directly into a logging file), but if dos2unix runs reliably in a deamon-like situation, it is certainly the better way.
EDIT: '\r' is ( char )13, my unwanted character, I see that now.
bperrybap:
just take Westfw's advice
Believe me, I have the utmost respect for his advice here and elsewhere, and I'm not discounting it one little bit. I WILL be holding his reply close to me. And until I can understand Jiggy-Ninja's I can't implement Jiggy-Ninja's at this very moment. But I do understand Westfw's, so I could implement that.
Delta_G, I believe I tried your suggestion as a first-try and the Serial class prevented it....? I'll retry when I get the opportunity...
Have to cut this short due to date with wife. I appreciate your help more than I can express!
kenneth558:
Understand that I don't want to categorically remove all blank lines from the cat STDOUT (which goes from there directly into a logging file), but if dos2unix runs reliably in a deamon-like situation, it is certainly the better way.
It isn't about sending/handling \r vs \n or thowing away blank lines.
It is about processing line endings when the input is considered "line" oriented.
i.e. what determines the end of a line of text
And line endings vary depending on OS.
CPM/DOS/Windows use \r\n (both)
Old MAC text files uses \r
New MACOS which is BSD unix uses \n but still supports the older MAC text files.
*nix uses \n
IMO, unix got it right and CPM/DOS/Windows and Apple made a mess of things.
That said, Windows, MAC, and *nix do have the ability to alter how they process line endings to be compatible with other systems.
linux already has the necessary tools to interpret data streams and delineate text lines the goofy way DOS defined them if that is needed.
One big question I have, is that if you are writing this data to log file, then it shouldn't really matter if you are using a decent tool to look at the log file since many tools on linux that use text files are smart enough to recognize the goofy DOS format with the extra carriage returns and deal with it.
So if you edit a DOS format file, you won't see phantom/extra blank lines and likewise if you cat a DOS format text file at the terminal window, you won't see phantom/extra blank lines.
That said, if you are inconsistent with your line endings you can confuse the code that auto detects the line
endings and end up getting phantom blank lines or seeing \r characters in your editor. So don't do that.
Like Robin2, I'm a bit puzzled as to how and why this is much of an issue.
Most of the issues I've had over the decades with respect to screwed up line endings is going the other way around as Windows tends be much less flexible.
It seems to be more of an issue of using the proper tool to get things done.
kenneth558:
And until I can understand Jiggy-Ninja's I can't implement Jiggy-Ninja's at this very moment. But I do understand Westfw's, so I could implement that.
What's to implement? I already did it for you. Just copy my function into your sketch and use it like I showed in the example. If you want you can get rid of the HardwareSerial& argument and just hardcode it to print to Serial.
Sterretje's and Westfw's suggestion is probably one of the worst ways to solve this problem. Modifying the core files will change the behavior of all your sketches, and will be wiped out the next time you update the core.
There are two sensible ways of solving this problem: a wrapper function in the Arduino sketch similar to what I did (prints the text, then prints just the LF), or changing the stuff on the Linux side to work properly when it receives CR-LF. Modifying the core files is an awful suggestion.
Actually there is another solution that could be done, and that is getting the Arduino developers to patch the Print class to allow selecting your EOL convention.
Jiggy-Ninja:
Actually there is another solution that could be done, and that is getting the Arduino developers to patch the Print class to allow selecting your EOL convention.
I tried getting them to address this for years and it went nowhere.
There have been some heated discussions on the mailing list about line endings and stream class buffering.
While getting changes into Arduino is much easier than before, they still tend to cling to a NIH (not invented here) mindset.
But even if you could get it in, it would be still problematic from a portability stance since it would require this new version of the Print class, and since it is down in the hardware core library instead of up in the common IDE bundled libraries directory, it means that it has to be implemented in every single core before it can be considered portable. That could take years to happen. i.e. Arduino can ship a new IDE with AVR bundled and fix DUE, but there are MANY other cores out there.
I also have tried several times to get a re-work of how the Arduino IDE handles its libraries which would make
moving things like the Print class up into the common libraries trivial, but so far, the IDE developers have not considered the existing issues large enough to warrant the work involved to make the changes.
My proposals included backward compatibility but so far i've not had any luck convincing them.
Sadly, over the years they have put more effort into propping up the existing library methodology than the effort it would have taken to really fix it.
It is so easy to deal with on the linux side, I'm not sure why anything else would be considered.
I do use the stty to Tx to the Arduino and the Linux version does allow the igncr flag BUT I'm not using stty for Rx - I'm using cat. I don't even know if I can run stty nohup and & (background daemon) as easily as the cat does it, and I'm taking your word for it that it works as deamon for Rx even while a separate process uses another instance of it to Tx.
I don't think you understand "stty" properly. "stty" is a tool for setting behavior of the underlying unix tty driver; you don't need it to be running concurrently with your "cat" of data from the serial port. If I'm sitting at a unix shell and type "stty igncr", the tty driver starts throwing away '\r' characters. If I type a "cat >myfile.dat" (terminated with ctrl-J, since "enter" normally generates '\r'), and then stream a bunch of data from an arduino with lines ending in "\r\n", you'll wind up with a file containing only the '\n' characters. (I regularly write little character-mode unix programs that I invoke with "stty raw;./myprog;stty cooked", because it's "easier" than figuring out how to get my program to set raw mode...)
Your big problem is wanting to use "cat" to collect data from a serial port. line endings are probably only the start of the problems you may experience. Various "terminal emulator" tools, with "capture mode" that have long since solved those problems, and you should probably look into using one of those.
westfw:
I don't think you understand "stty" properly. "stty" is a tool for setting behavior of the underlying unix tty driver; you don't need it to be running concurrently with your "cat" of data from the serial port.
Is it even possible to run stty concurrently with cat on a serial port? i.e. if you run stty in the background, won't it just complete and exit?