Potential Bug: Serial.ReadStringUntil

Forgive me if this is a repost, I did a search and didn't find any related information on if this is a known issue.

Take the Arduino program below (I ran it on Uno). If I type a really really long string like

"12345678901234567890123456789012345678901234567890123456789012345678901234567890"

Everything works fine as everything is done in 1 pass

However if I type a long string like:

"Hi there, how are you doing? I would really like this tokenizer to properly work, but it's giving me a fit and scrambling random letters and words. What is up with that?"

You'll see the output is garbled with

Token: 'Hi'
Token: 'there,'
Token: 'how'
Token: 'are'
Token: 'you'
Token: 'doing?'
Enter a line of text:
Token: 'I'
Token: 'would'
Token: 'really'
Token: 'like'
Token: 'this'
Token: 'tokenizer'
Token: 'to'
Token: 'properly'
Token: 'work,'
Token: 'but'
Token: 'it's'
Token: 'giving'
Token: 'me'
Token: 'a'
Token: 'fit'
Token: 'and'
Token: 'scrambndom'
Token: 'letteds.up'
Token: 'with'
Token: 't'

I'm using the Serial Monitor and the latest versions of libs and Arduino IDE.

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  Serial.setTimeout(50);
}

void loop() {
  // put your main code here, to run repeatedly:
  Serial.println("Enter a line of text:");

  readLine();
}

void readLine()
{
  bool waitingForData = true;

  while (waitingForData)
  {
    if (Serial.available())
    {
      String token = "";
      waitingForData = false;

      do
      {
        token = Serial.readStringUntil(' ');
        if (token != "")
        {
          Serial.print("Token: '");
          Serial.print(token);
          Serial.println("'");
        }
      } while (token != "" || Serial.available());
    }
  }
}

memory Kalashnikov :slight_smile:

The fault is somewhere in the serial read buffer is implemented. This is disappointing at this would fail qualification testing for a large scale industrial purpose. Is there no hardware flow control?

volatile bool doneLine = true;
String token = "";

void setup() {
  Serial.begin(9600);

}

void loop() {

  if (doneLine)
  {
       Serial.println("Enter a line of text:");
       doneLine = false;
  }
}

void serialEvent()
{
  //Serial.println("Serial Event");
  while (Serial.available() > 0)
  {
    //Serial.print("Available() = ");
    //Serial.println(Serial.available());
    byte value = Serial.read();
    char inChar = (char)value;

    if (inChar != '\n' && inChar != ' ' && inChar != '\r')
    {      
      token += inChar;      
      Serial.print(value);
      Serial.print(' ');
    }
    else
    {
      while (Serial.availableForWrite() < token.length() + 16)
      {
        delay (100);
      }
      Serial.println("");
      Serial.print("Token: '");
      Serial.print(token);
      Serial.println("'");
      token = "";

      if (inChar == '\n' || inChar == '\r')
      {
         doneLine = true;
      }      
    }
  }
}

If you want reliability, don't use String (capital S).

Read Robin's updated Serial Input Basics to get ideas.

Note
Serial is slowwwwwwww

Serial.write waits until there is a space in the buffer. your delay doesn't do good. you have 9600 baud for write too, not only for read. until you wait for sending, you lose bytes on input. and you send much more then you receive

dth1971:
Is there no hardware flow control?

Normally with Serial, there is no flow control. Unless you also connect RTS & CTS signals, and also use software on the board which supports those signals, ordinary serial doesn't have any flow control.

However, native USB serial does have flow control, where you're using a USB port directly and not communicating actual serial to another chip with does usb-serial for you. So if you use MRK1000 or Leonardo/Micro, or the native serial port on Zero or Due, or Teensy (which I make) or others with native USB, then you do have flow control built in.