Serial Text I/O for the Real World

V3.0.0 of the SafeString library (available from the Arduino Library manager) includes a replacement for Arduino Serial Text I/O that works in real world applications. A detailed tutorial is here

The Quick start is: -

  • Install the SafeString library (V3 +) from the Arduino Library Manager or the zip file
  • For text input use a SafeStringReader - a non-blocking robust high level reader for reading lines of data or general text with delimiters, with optional echo and optional non-blocking timeout
  • For text processing use the SafeString methods - what Arduino Strings was meant to be, but with out the memory problems and odd errors and with detailed error messages.
  • For text output write to a BufferedOutput - the non-blocking replacement for Serial print, so your debug msgs and results do not delay the rest of the loop and cause missed input.
  • For testing use a SafeStringStream - stress tests your sketch's text processing.
  • If you need a larger input buffer use a BufferedInput - for when the sketch processing delays the reading of the input and chars are lost

Arduino Serial is not suitable for handling Text I/O in Real World applications.

You should avoid using any Serial methods in your loop() code and any code it calls. This is because almost all Serial methods will block the rest of your loop() code from running so you will miss real world inputs and your outputs will be delayed. Print() statements are the primary means of debugging Arduino sketches, but adding Serial.print()s adds more delays and often causes more problems then they find.

The tutorial covers reading text input, parsing for commands, parsing text to numbers, e.g. GPS data, and outputting results and debugging messages. All without blocking the rest of the sketch so that your Arduino won't miss the real world inputs and be able to reliably control real world devices, stepper motors, relays etc.
For a stepper motor example see Simple Multitasking in Arduino

SafeStringReader provides a high level non-blocking read() method that simplifies handling Serial Text input and returns the results in SafeStrings so avoiding the user programming errors that are common when working with char[]s

For example this code recognizes two user commands from an unlimited length input stream, using only two small SafeStrings. It can be easily extended to handle more/different commands and numeric input. See the stepper motor example in Simple Multitasking in Arduino.

#include "SafeStringReader.h"

// create an sfReader instance of SafeStringReader class
// that will handle commands upto 5 chars long
// delimited by space, comma or CarrageReturn or NewLine
// the createSafeStringReader( ) macro creates both the SafeStringReader (sfReader) and the necessary SafeString that holds input chars until a delimiter is found
// args are (ReaderInstanceName, expectedMaxCmdLength, delimiters)
createSafeStringReader(sfReader, 5, " ,\r\n");

void setup() {
  Serial.begin(9600);
  for (int i = 10; i > 0; i--) { // pause a little to give you time to open the Arduino monitor
    Serial.print(i); Serial.print(' '); delay(500);
  }
  Serial.println();
  SafeString::setOutput(Serial); // enable error messages and SafeString.debug() output to be sent to Serial
  sfReader.connect(Serial); // where SafeStringReader will read from
  sfReader.echoOn(); // echo back all input, by default echo is off
  sfReader.setTimeout(2000); // 2000mS = 2sec timeout
}

void handleStartCmd() {
  // handle start cmd here
}
void handleStopCmd() {
  // handle stop cmd here
}

void loop() {
  if (sfReader.read()) {
    if (sfReader == "start") {
      handleStartCmd();
    } else if (sfReader == "stop") {
      handleStopCmd();
    } // else ignore unrecognized command
  } // else no delimited command yet

  // rest of code here is executed while the user typing in commands
  }
}

SafeStringReader has options for echo and timeout.

The BufferedOutput class included in V2.0.6 provides non-blocking Serial Text Output. A practical example of processing GPS data is provided which overcomes the problem of missing part of the GPS messages while handling user input and outputting debugging and other messages.
Typical use is

#include "BufferedOutput.h"
createBufferedOutput(output, 66, DROP_UNTIL_EMPTY); // modes are DROP_UNTIL_EMPTY, DROP_IF_FULL or BLOCK_IF_FULL

void setup() {
  Serial.begin(BAUD_RATE);
  output.connect(Serial);  // <<<<< connect the buffered output to Serial 
}

void loop() {
  output.nextByteOut(); // <<<<<<<<<< need to call this each loop to release next bytes from buffer
  . . . 
  // replace Serial.print() with output.print() in the loop() code
}

Finally the SafeStringStream class provides a simple means of testing of sketches that require text input without having to type in the input every time the sketch is run.

No thanks. I already wrote my own library that handles actual real world text I/O very well. It is completely non-blocking. The token handling is intriguing, though. I would usually read in an entire line and then parse it.

Also, just in future FYI this belongs in the "Showcase" forum, not "programming questions".

Arduino Serial is not suitable for handling Text I/O in Real World applications

that is abusive generalization. read() and available() are totally doing the job. I would suggest to study Serial Input Basics to handle this and be self sufficient with standard C libraries that any real world developer needs to know.

Also understanding how this works will help any real world developer handle Serial binary protocols. The underlying use of cString (through the class) rather than uint8_t buffer probably makes it hard to deal with a protocol starting, including or ending with null byte '\0' for example

See discussion and opinions on this library here.

@drmpf

Although I believe all initiatives to share code is great, Can you clarify what this means on the corporate web site you are pointing to?

© Forward Computing and Control Pty. Ltd. NSW Australia
All rights reserved.

If all rights are reserved - what does it have to do on a forum where open source is the norm? Any chance to get your company to consider adding a typical MIT or GNU license disclaimer or equivalent?

———
As a side personal note I prefer tutorials which do not introduce dependencies on other libraries (such as your SafeString library) that are not standard as it adds complexity.

For real world IO on Serial, I would recommend Serial Input Basics to handle this as it demonstrates all best practices that can be applied with standard functions for both an ASCII or Binary based asynchronous protocol and equips real world programmers with the knowledge they need to have for memory management.

Hi J-M-J,
The problem is that Arduino is not targeted at real world developers, but at users just starting out.
Even very experienced programmers make coding errors with char[] and c-string methods
See the latest IPhone buffer overflow bug for just one example. The tutorial has more references.

I think we can do better for new Arduino users.

The SafeString library is my offering. As well as handling input is also covers non-blocking serial text output which is the main (only?) means of debugging Arduino sketches and when working with real world, real time applications. Serial.print()'s just make things worse.

Hi J-M-J,
for the software license for the SafeString library, check the top of the various files. Basically it is use at your own risk. The copyright your refer to is for the webpage tutorial as a publication.

I have two problems with Serial Input Basics for Serial Text Input/Output. (SafeString does not cover binary data)

1 Serial Input Basics does not actually cover non-blocking Serial Output.
Serial Text Output is the main (only?) means of debugging Arduino sketches, but in real world applications, add blocking Serial.print() statements causes more problems than they find. The non-blocking BufferedOutput class in the SafeString library overcomes this problem

2 Serial Input Basics is basic low level code. It works well if you code it well, but even experienced coders continue to make mistakes. I don't believe it should be recommended, in the first instance, to Arduino beginners when there is a robust, high level alternative that avoids char[]s and c-string functions. i.e. SafeString.

The Arduino Serial library tried to add some 'useful' higher level methods, like find() readUntil() and readString()
However these are correctly advised against by Serial Input Basics, because they can block the rest of the loop.

The SafeString methods are all non-blocking and I believe provide a practical realization of the sort of user friendly functionality that Arduino was trying to provide, and failed.

I tend to disagree - Arduino targets all sorts of developers, from beginners to pros. It seems your company is using it in a pro capacity for example.

Yes, fully agree. Even very experienced programmers make coding errors (your library does not address checking binary buffer overflow operations though).

When it comes to true safeStrings, I would think of this Intel/Cisco initiative.

As expressed before the reason I won't recommend this library is that it's not so safer than using the standard C functions you used to develop it. In case of text string, whilst you won't crash directly during the string operation, unless the developer has coded the buffer overflow check in the first place the app will be left working on a limb and will crash/misbehave probably a few lines later. And if you check for buffer overflow, well then you've done the work that was required and don't need to do it again in the library.

I think most developers with a bit of practice have come up with their own versions of cString based expanded capabilities.

What beginners needed is what the String class provided for - dynamically growing text strings (which come with their own issues) and for more advanced C/C++ developer in the real world, they should know the default cString capabilities in the first place. I actually believe that using the String class with promoting the use of "reserve()" kinda matches what you are bringing to the table. It solves 99% of the poking holes in the heap issue whilst offering the safety belt of dynamic expansion if required.

Now - as I said before, all code sharing should be encouraged. For example there is value in the helper functions you provided and as general example of what a startsWith() or startsWithIgnoreCase() etc could look like.

Thanks for the answer - yes I had seen the mention in the library itself. I was just suggesting to have something similar for the codes provided on the web pages (if they are the same examples provided in the libraries then you can disregard my comment).

I agree with #1 - indeed it can become an issue and it's important to pinpoint that this can happen.

Your way to solve it basically means a choice of loosing debug data versus blocking the loop. This is also at the expense of more memory used (new buffer allocated), which is often an issue on small micro-processors. So, does it fully solve the issue — No — but is it useful to understand what's happening — Yes for sure.

if you don't want to block, your debug statement could be wrapped in a simple testif (Serial.availableForWrite() > 15) Serial.println(F("Debug message"));which would basically drop the log if it can't be added (that could be made into a DEBUGPRINT macro so that 15 is calculated)

I disagree with your view on #2, as expressed elsewhere SafeString protects against buffer overflow but just postpones the issues as the beginners and "experienced coders who continue to make mistakes" will be left with the wrong content in their string if they did not code the bound checks in the first place. (and if they do it in the first place, then there is no need to do it again in the library).

I also agree with the risks associated with find() readUntil() and readString()... I don't recommend those either.

Overall, Don't get me wrong - I'm not saying your code or ideas are not good, it is I'm sure.

My objection is more fundamentally in a learning/educational process "for the real world": you are better off investing the time (as you probably did before writing this library) in understanding how memory works, the cString libraries and their shortcomings or just use the String class with reserve() if you are a true beginner and don't want to invest any time at all.

but that's just my view - other might see it differently.

J-M-L:
if you don't want to block, your debug statement could be wrapped in a simple test
if (Serial.availableForWrite() > 15) Serial.println(F("Debug message"));
which would basically drop the log if it can't be added (that could be made into a DEBUGPRINT macro so that 15 is calculated)

Unfortunately that is not a solution to blocking debugging messages.

If you don't see the "Debug message" you don't know if the sketch did not take that branch
OR if it did but the message was dropped due to lack of space

Also availableForWrite() is not a reliable test.
ESP32 and ESP8266 still blocks when availableForWrite() == 1,
Arduino NanoBLE always returns 0 for availableForWrite()
and on some boards Serial does not extend from HardwareSerial and so availableForWrite() may not exist.

The BufferedOutput class in the SafeString library handles all of these cases.
As detailed in the Serial Text I/O for the Real World tutorial, if you use BufferedOutput, your loop() will not be blocked and often a small BufferedOutput buffer, a higher baud rate and abbreviated messages show all your debug msgs.

If any chars do get dropped then you will see ~~ at the end of the line so you know when something is missing.
The tutorial covers what to do in those cases.

I agree with the fact that your approach is one possible techniques that would work but is not the only one and your mileage may vary based on what you need to do. Allocating an extra 64 byte buffer might actually crash the program if you are low in SRAM...

Sometimes you don't want to wait for many loop() cycles to see all your data out... Sometimes the blocking nature of the Serial port (when timing is not an issue) is actually very helpful to find where the issue in the code is.

Assume you have a debugPrintln() macro that print and flush. You would know which function is causing an issue with such a code:

void loop()
{
  debugPrintln(F("0 - veryyyyyyyyyy looooooooong debug text..................."));
  instruction1();
  debugPrintln(F("1 - veryyyyyyyyyy looooooooong debug text..................."));
  instruction2();
  debugPrintln(F("2 - veryyyyyyyyyy looooooooong debug text..................."));
  instruction3();
  debugPrintln(F("3 - veryyyyyyyyyy looooooooong debug text..................."));
  instruction4();
  debugPrintln(F("4 - veryyyyyyyyyy looooooooong debug text..................."));
  instruction5();
  debugPrintln(F("5 - veryyyyyyyyyy looooooooong debug text..................."));
}

some feedback on your other points

drmpf:
If you don't see the "Debug message" you don't know if the sketch did not take that branch
OR if it did but the message was dropped due to lack of space

Indeed - this is a tradeoff, you have some in your setup as well with DROP_UNTIL_EMPTY, DROP_IF_FULL or BLOCK_IF_FULL. and if you see many ~ you just don't know what happened... not very helpful in my opinion. Also if your code crash, printing one char per loop cycle means you have way less information available to debug than if you had given a chance for the hardware to print as much as possible before crashing.

drmpf:
Also availableForWrite() is not a reliable test.
ESP32 and ESP8266 still blocks when availableForWrite() == 1,

you would usually output more than 1 byte for debug... (at least the '\n' if you separate output)

On an ESP32 it would not be the right approach, it's a multi-cpu chip, so what you get as result and reality when you start writing could be very different. The fifo buffer might have cleared a bit or another thread might have started writing to the UART and you actually have less space. (On an ESP32 the code writes directly to the hardware so it's different than the ring buffer of the ESP8286 for example)

drmpf:
Arduino NanoBLE always returns 0 for availableForWrite() and on some boards Serial does not extend from HardwareSerial and so availableForWrite() may not exist.

Fair enough, availableForWrite was a late addition on ESP for example. but why would you use Stone Age debugging techniques on many of those boards when you can use JTAG and hook up J-Link Debug Probes and use GDB directly to the Hardware? (eg Arduino Nano IOT or ESP32).

PS: anyway, we are too far from OP's request.

those interested in your libraries should use them, can't hurt if that meet their needs.

Yes getting way off the point of the topic here.
Can the moderator move our replies to the Serial Text I/O for the Real World topic.

Topics split, moved and merged as requested

Hi
J-M-J now that we are on the appropriate topic, so comments on you last post.
Not sure what the point of you example code was?
The application I was thinking of was real world interactions where you cannot afford to print and flush, because nothing works if you do that.
A case in point. A relative of mine was building an exhibit for the Boston Science Museum timing time of flight of a ball. When he added some debugging output the loop missed the beam break. Now I know perhaps he could have used an interrupt and low level timer register controls to get the job done, but the code he had was workable, just not when he added Serial.print()s

Allocating an extra 64 byte buffer might actually crash the program if you are low in SRAM...

I take care to do the allocation globally so that it is clear to the user when they compile just how much SRAM they have left. On the Uno/Mega at least you get a warning if it is low.

Sometimes the blocking nature of the Serial port

Yes I agree, that is why there is a BLOCK_IF_FULL mode. But you still get the benefit of a larger buffer.

Also if your code crash, printing one char per loop cycle means you have way less information available to debug than if you had given a chance for the hardware to print as much as possible before crashing.

A few points here. If you use SafeStrings you are much less likely to have your program 'crash' Also nextByteOut() will actually fill the HardwareSerial Tx buffer with with as many bytes are there is room. Finally if you do code a crash, then you can always call flush() at appropriate points to force the BufferedOutput to be completely written (a blocking call)

Why would you use Stone Age debugging techniques on many of those boards when you can use JTAG and hook up J-Link Debug Probes and use GDB directly to the Hardware

Not something I would expect a new or hobbyist user to use. I looked at it when I was doing my BLE low power projects and was put off by the added expense and learning curve.
That form of debugging does not seem to be encouraged by Arduino.

I am interest though so can you point me to a detailed tutorial on it aimed at Arduino users.

OK I guess we have different understanding of "real world". For me it's no longer in beginner hobbyist territory and starts to be more on the pro side (even if done by a hobbyist). I'd expect "pro" (complex) work to be done the right way with some understanding of what's going on.

If you use SafeStrings you are much less likely to have your program 'crash'

Yes definitely, it's less likely to crash on the actual string buffer overflow (actually it won't crash there thanks to your work). But if you don't trap the buffer overflow in the code then you are left with an incomplete buffer which is likely to mean the rest of the code will misbehave (if the developer did not implement error checks, s/he probably did not ensure parsing the buffer was done right either).

On SafeString : My view is that if I need to worry that a string operation did not succeed by checking the error status of my operation AFTER the operation has been attempted (so that I take the necessary steps for the real world), I don't see what's really the gain compared to doing so BEFORE the operation using the standard cStrings functions when needed. And if I have sized all my buffers appropriately to never run into such an overflow, then I don't need to pay the cost of the checks every time I do an operation. So I don't see where there is an obvious win there in real world where resources are scarce.

Compare this to the String class as provided within the IDE. I can size the buffers appropriately (reserve()) to mitigate the risk of poking holes in the heap - same size you would use in SafeString - but if something bad happens then I'm still left with a fully working String that did not drop anything. That's hobbyist friendly and more forgiving.

There are multiple ways to skin a cat (hate this expression), I see you are sharing a set of libraries with their interdependencies and so embracing one (buffered output) means embracing the other (SafeString) and a way to architect the code (call output.nextByteOut() ) and I see how there is value to you to buildup on top of well tested code. It's a good engineering practice, I'm sure it has its use case and could be enforced as a standard in a company for example, but I don't see enough value add to relinquish fine grain control in my code on tiny MCU where RAM and CPU time are heavily constrained to embrace something with larger ramifications.

In a nutshell, I find the String class good enough for hobbyist and the cStrings functions (or intel/cisco SafeStrings) in all their glory and tradeoffs better suited for 'real world' (pro like) work and so I see limited value in your library for MY case. Other may see it differently, fully respect the view and I'm sure that will be helpful to some.

Check out this post for the problems Arduino users get into using Stirngs and c-string constructs.

The code is correct and well formed but just not usable in the real world. It was much easier to fix this with SafeStrings then to try and work around all the problems with the existing code.

The end result for this Arduino user is very impressive. Voice controlled access to EV stats and home heating settings.
He appears to be well versed in C and programming, just not 'pro' enough to get it working.

As a separate issue the OBD library he is using has a buffer overflow error when the OBD message buffer fills up.
It has not bitten anyone yet and may well go years before un-expectedly crashing a user's program.

As you know this type of error is not uncommon for profession programmers, which is why I don't believe we should be promoting using char[] to Arduino users.

You mention the problem of a SafeString program 'failing later' if it not sized correctly.
These failures do not crash the program and it is a simple step to turn the error messages on and find exactly were the problem it and fix it.
Not so easy for buffer overflow on Arduino UNO. It would be nice to have a stack trace like the ESP32 to work with.

Sure SafeStrings does not solve all the possible buffer crashes and user still need to made aware of the issues, but with the existence of standard c-string methods and the String class, users tend to get into more problems doing string processing then other buffer stuff.

I’m not sure what’s your point with example from bad work from developers. I could give you examples that are perfectly fine with cStrings...Your library is such an example showing that cStrings used the right way do work totally fine...

Is Your point why should they reinvent the wheel since you have done the work?

I believe every C programmer should understand cStrings in details as well as Associated functions and practice with them (like I’m sure you did). It’s a great learning they can use everywhere and write their own helper functions if they wish so (which is also a good exercise). I’ve seen dozens of similar work throughout the years.

I’m getting tired of this discussion. My point is that I don’t see enough value add in real life and I think it works against learning the fundamentals and thus would not recommend it... but that’s OK, you’ll find other people who will love it I’m sure.

A safer string library is desirable, especially for beginners, but it needs to be very easy to use. I have a hard time evaluating how "easy to use for beginners" libraries are these days. :frowning:
A set of non-blocking, less-primitive Serial input routines is useful. The lack of a readline()-like API for Arduino leads to a significant amount of frustration, and is pretty much directly responsible for many of the incorrect and/or sub-optimal uses of both C strings and the Arduino C++ Strings (Buffer += Serial.read(); Shudder.)
A non-blocking Serial write is a lot more ambiguous, especially with the requirements needed to implement it. :frowning:

Although - someone mentioned cisco... cisco code had rather a lot of bugs caused by the fact that their printf()-like APIs could block, and things would change out underneath. But those were more due to the need for locking of the things being printed, than the blocking of print itself...

Just writing up V3 of SafeStrings that includes SafeStringReader that does a very simple line read

// create an sfReader instance of SafeStringReader class
// that will handle input lines upto 80 chars long terminated by newline
// the createSafeStringReader( ) macro creates both the SafeStringReader (sfReader) and the necessary SafeString that holds input chars until a delimiter is found
// args are (ReaderInstanceName, expectedMaxInputLength, delimiters)
createSafeStringReader(sfReader, 80, '\n');

void setup() {
   Serial.begin(9600);
   sfReader.connect(Serial);
   sfReader.echoOn();   // optional for echo
}

void loop() {
   if (sfReader.read()) {  // non-blocking
      // sfReader now has the whole input line less the \n
      ... 
    }
  }

If the input line length exceeds 80, the line is ignored and the input skips to next delimiter (\n in this case). If you have SafeString error msgs turned on then you will also get an error message

 sfReader -- Input exceeded buffer size. Skipping Input upto next delimiter.

You can also specify a non-blocking timeout if the delimiter is missing.
sfReader.setTimeout(2000); // 2sec timeout no chars and no newline

You can set other delimiters to parse user commands etc.