Serial has no terminator, so use a word for one?

I'm using an arduino uno to read a string of text data from the pc, split the string and print parts of it to different areas on a 24 x 2 character lcd.

A typical string would be :
1,76,2,0,15:23,- 8.3,E.-dorf Krkhaus @076 02 107 24 B ,24,107,1,Krankenhaus,noch nicht mit leben gefüllt

The text in the string changes all the time and the length of the string alters, sometimes parts of the string are totally empty between commas, like :
1,0,0,0,15:20,.,,0,0,-1,,noch nicht mit leben gefüllt

The problem i have is there is no terminating character to indicate the end of a string, but the last 4 words are always there, with the very last word never being found anywhere else in the string.

So i want to search the incoming string for the word gefüllt, and use that as the terminator character.
But i'm not sure how i would go about this,

i am using the dreaded String method for now, sorry, i will use an alternative to strings when i get the sketch doing what i want, i seem unable to absorb the programming stuff easily, so i can just about get my head around strings.

A string has a definite length. You seem to be describing a continuous stream of data, not a string. Are you sure there is no CR/LF at the end of each "string"? Where does the data originate? A file, etc?

Paul

gazz292:
The problem i have is there is no terminating character to indicate the end of a string, but the last 4 words are always there, with the very last word never being found anywhere else in the string.

I would use a search for a word as a last resort - it will involve a lot of code. If the 3 letters "llt" are unique that would simplify things.

What is the baud rate? If it is low (not exceeding 115200, perhaps) I would try using a comma as the end-marker.

Is there an obvious gap between messages - if so the time between characters could be used to define the end of a message?

How long is the longest message?

...R

PS ... to detect a sequence of characters you need to look for the first character (G in your case). When that is detected you need to see if the next characters is as expected (E in your case). If it is not E then go back to waiting for a G, or if it is a G then check if the next character is E. It gets very tedious with 7 characters (or is it 8 with the umlaut).

Sorry, i am useless at software things, i'm a mechanical type of guy, i'm learning to TIG weld atm and i have picked that up way easier than i learnt a tiny portion of how to use the dreaded Strings with the Arduino.

So, it's a data stream of text? (would that be a packet of data?) separated by commas, so CSV's.

What happens is:
A bus simulator game is running on the PC, and a communications interface runs also on the same pc which reads variables in the game and sends them out over the serial ports at a fixed speed of 115200,
Then arduino's receive and use the data in the streams / packets.

Unfortunately, the person who wrote the communications interface has moved onto other things, so the issues with it are stuck, like data stream 5 isnt even active, there's no terminator on stream 3, that text ''noch nicht mit leben gefüllt'' is German for 'not yet filled with life' and is supposed to be where the bus interior temperature would go and so on.

There are a total of 5 'data streams' on 5 (usb) serial ports (4 working)
The one i'm interested in is for the IBIS lcd (IBIS = 'Integrated Bus Information System' it runs the passenger info displays, makes announcements, monitors if running late etc)
That stream or packet is plain text, and unfortunately it does not have any terminator character at all, things like \r, \n, do nothing, there's no special terminator character like there is on some of the other data streams.

There is a very clunky way of changing that text in the last string to a ';' but it needs doing every time the bus sim is started, involves using the task manager and is a royal pain in the ass, not something i want to do when i am building a replic bus drivers cab, and want to press a button that starts the pc up, hides windows and loads straight into the simulator with the communications program running and arduino's working.

The one good this is that data is on com3 i set to be 'refreshed' every 200 milliseconds (that is the one variable that can be changed easily)

The arduino reads this data stream, then separates it into 'strings' where the commas are,

These strings are then used to write text in certain places on the lcd.

For my use i only need the 6th and 7th string, so for this, one 'packet' of data that would be :
1,76,2,0,15:23,- 8.3,E.-dorf Krkhaus @076 02 107 24 B ,24,107,1,Krankenhaus,noch nicht mit leben gefüllt
I just need to read and use '- 8.3' and 'E.-dorf Krkhaus @076 02 107 24 B'
That data will change as the next bus stop name changes and the delay time changes, or other things are shown on the IBIS lcd, like setting it up.

Here's my code i use atm, i need all the comments in it to help me figure out what's going on, i may have some functions commented wrong, and i know Strings are bad, but i want to get this code working properly before working on better methods of doing this, and try to learn was i go, and if i write the code i can hopefully change it myself if requirements change.

/*  Text to use with serial monitor to test , a typical output from the 'IBIS' com port with everything set up on the IBIS: 
 *  1,76,1,0,14:36,- 4.2,E.-dorf Krkhaus     @076   01 105    6 A ,6,105,0,Krankenhaus,noch nicht mit leben gefüllt,
 *  
 *  Output when bus electricity is off, and hence IBIS display is blank / off: 
 *  1,0,0,0,12:45,.,,0,0,-1,,noch nicht mit leben gefüllt,  
 *  
 *  I plan on checking for empty 'strings' later, as an empty 'delay' string currently prints weird characters on the LCD.
 *  And turning the bus electric off leaves the last text on the LCD instead of clearing it */

#include <LiquidCrystal.h> // use lcd library

LiquidCrystal lcd(12, 11, 5, 4, 3, 2); // lcd pins

String inputString = "";  // string to hold incoming csv's
String split_lines = "";  // string to hold data for splitting the lines on lcd
boolean stringComplete = false; // is the string complete, at this point - no?

// Set up string index, comments are What the individual strings mean
String text1; // '1' Delay yes / no
String text2; // '76' Bus line number - 3 digits
String text3; // '1' Route
String text4; // '0' Bus stop index
String text5; // '14:36' Time
String text6; // '- 4.2' Delay in minutes and 10ths of a minute, '-xx.x' is early, '+xx.x' is late, '  0.0' is on time
String text7; // 'E.-dorf Krkhaus     @076   01 105    6 A ' Bus stop name, i have changed this with a bodge in Komsi to output 'IBIS lcd contents' 2 lines seperated by @
String text8; // '6' Zone
String text9; // '105' Destination
String text10; // '0' Direction (0 = A, 1 = B, -1 = not set)
String text11; // 'Krankenhaus' Line number - upto 5 digits, another bodge in komsi makes this 'interior display text'
String text12; // 'noch nicht mit leben gefüllt' Interior temp, not working so it displays that text, translated from German =  'not yet filled with life'


void setup() {
  Serial.begin(115200); // start serial coms at the speed indicated
  inputString.reserve(200);  // reserve 200 bytes for data in string
  lcd.begin(24, 2); // start using lcd, set lcd paramiters, number of colums, rows.
}

void loop() {

  if (stringComplete) {  // run the following code

    String split = inputString; // split up the line of text from serial data stream
    text6 = getValue(split, ',', 5); // get data between commas from string 6, which is for delay
    text7 = getValue(split, ',', 6); // get data between commas from string 7, which i've set to IBIS screen contents
    inputString = ""; // place to store string data
    stringComplete = false; // string is still not yet complete

    String line1, line2; //String for IBIS text to be split in 2 lines    {
      split_lines = text7; // Split off the ibis text line
      int a = split_lines.indexOf('@'); // Seperator character for the 2 lines
      if (a >= 0) { // If text in the buffer, split it
        line1 = split_lines.substring(0, a); // substring for line1, text befoer the @
        line2 = split_lines.substring(a + 1); // substring for line2, text after the @        
      }


// start printing text to the LCD
    lcd.setCursor(0, 0); // Sets cursor on the LCD, column, row.
    lcd.print(line1); // Displays the first line of text 
    lcd.setCursor(0, 1); 
    lcd.print(line2); // Displays last line of text
    lcd.setCursor(21, 0); // move cursor here, print the delay  -, or + symbol above the numbers as on the real IBIS.
    lcd.print(text6.charAt(0)); // print the +, -, or nothing, for running late, running early, on time.
    lcd.setCursor(20, 1); //bottom row, 21st column
    lcd.print(text6.charAt(1)); // first number of delay time, 10's of minutes
    lcd.setCursor(21, 1);
    lcd.print(text6.charAt(2)); // 2nd number, 1's of minutes
    lcd.setCursor(22, 1);
    lcd.print(text6.charAt(3)); // 3rd number.. actually a dot.
    lcd.setCursor(23, 1);
    lcd.print(text6.charAt(4)); // final number, tenths of a minute, i.e. 0 to 60 seconds but shown with with numbers 0 to 9... metric time?! 
    // i know i could do the above an easier way, something about read everything from the 1st character to the end of that string?
  }
}


void serialEvent() {
  while (Serial.available()) { // Only do things when serial is active
    char inChar = (char)Serial.read(); //read the serial on charecter at a time?
    inputString += inChar; // Add the characters together to make the stirng?
    if (inChar == ';') { //new line charecter, if seen means string is complete.. dosent exist on this data stream, so i clunkilly change it to ';' every time i start the bus sim, i want to avoid this.
      stringComplete = true; // Tells the first bit of code in loop to process the data now
    }
  }
}


String getValue(String data, char separator, int index) // stuff for splitting strings for the index and that?.
{
  int found = 0;
  int strIndex[] = {0, -1};
  int maxIndex = data.length() - 1; // not sure

  for (int i = 0; i <= maxIndex && found <= index; i++) {
    if (data.charAt(i) == separator || i == maxIndex) {
      found++;
      strIndex[0] = strIndex[1] + 1;
      strIndex[1] = (i == maxIndex) ? i + 1 : i;
    }
  }

  return found > index ? data.substring(strIndex[0], strIndex[1]) : "";
}

Debugging this project will be difficult on an Uno as it has only one HardwareSerial port and SoftwareSerial can't run at 115200 baud. If you have a Mega and a USB-TTL cable you could set up two separate channels of communication with the PC - one to receive the data and the other to display what has been received on the Serial Monitor. The Serial Monitor will be much more convenient than an LCD because you can scroll back to view earlier lines of output.

If you have no choice but to do everything on the Uno then I suggest you start by sending a short message with the desired end-text and develop a program along the lines I suggested at the end of Reply #2 to detect the end and display the message when it has all arrived.

Have a look at Serial Input Basics. You could use the second example as a starting point.

...R

The one good this is that data is on com3 i set to be 'refreshed' every 200 milliseconds (that is the one variable that can be changed easily)

Make use of the time between messages.

Add the below near the top of your code.

// last time that a character was received
uint32_t lastReceivedTime;
// message timeout; adjust to needs
uint32_t messageTimeout = 50;

Best place is probably just after below so everything related to the communication is together

String inputString = "";  // string to hold incoming csv's
String split_lines = "";  // string to hold data for splitting the lines on lcd
boolean stringComplete = false; // is the string complete, at this point - no?

And modify your serialEvent as shown below

void serialEvent() {
  while (Serial.available()) { // Only do things when serial is active

    // remember the time that we received something
    lastReceivedTime = millis();

    char inChar = (char)Serial.read(); //read the serial on charecter at a time?
    inputString += inChar; // Add the characters together to make the stirng?
  }

  // check for timeout if inputString is not empty
  if (inputString != "")
  {
    if (millis() - lastReceivedTime >= messageTimeout)
    {
      stringComplete = true; // Tells the first bit of code in loop to process the data now
    }
  }
}

I've set the message timeout to 50 ms; you can make it shorter if it needs to be more responsive. It needs to be shorter than the time between messages.

Code compiles but not tested.

Note that the use of String (capital S) can result in random problems at run time; stay away from it unless you know exactly how it works and what the implications can be.

Based on Robin's receiveWithEndmarker, below an example that uses a timeout instead of an end marker. Again it does compile but is not tested.

const byte numChars = 32;
char receivedChars[numChars];   // an array to store the received data

boolean newData = false;

// last time that a character was received
uint32_t lastReceivedTime;
// message timeout; adjust to needs
uint32_t messageTimeout = 50;


void setup() {
  Serial.begin(9600);
  Serial.println("<Arduino is ready>");
}

void loop() {
  recvWithTimeout();
  showNewData();
}

void recvWithTimeout() {
  static byte ndx = 0;
  char rc;

  // if at least one character was received and we did not receive characters for a given time, assume that the message is complete
  if (ndx != 0 && (millis() - lastReceivedTime >= messageTimeout))
  {
    receivedChars[ndx] = '\0'; // terminate the string
    ndx = 0;
    newData = true;
    return;
  }


  while (Serial.available() > 0 && newData == false) {

    // remember the time that we received something
    lastReceivedTime = millis();

    rc = Serial.read();

    receivedChars[ndx] = rc;
    ndx++;
    if (ndx >= numChars) {
      ndx = numChars - 1;
    }
  }
}

void showNewData() {
  if (newData == true) {
    Serial.print("This just in ... ");
    Serial.println(receivedChars);
    newData = false;
  }
}

Note: adjust numChars to your needs.

1 Like

brilliant, thankyou everyone,

i do have a couple of mega's here, and i'm sure i have a usb ftdi converter somewhere, i got it for programming STM32 boards for a force feedback steeringwheel project, that i havent even started yet.. but that's just a copy and paste job to program, but maybe that will work with the mega,

i have an Arduino due, with the 2 usb ports, but i know one is supposed to be only a programming port, and als oit uses some 'sam' chip, so possibly different to program, and the end code will run on an uno or maybe even a pro micro if possible.

I shall try the suggested bits of code, and also try to work my way through writing the code all my self bit by bit, and get parts working and move on, tho i am sure i still wont understand it all at the end :smiley:

It would all be so much easier if that communications program had that little terminator character programmed in for the ibis output,
but short of decompiling and altering that program (which i wouldn't do as it's not my code / program, not that i'd even know whee to start hacking things like that) i don't think that's going to happen.

sterretje:
Based on Robin's receiveWithEndmarker, below an example that uses a timeout instead of an end marker.

Thanks for that. I will bookmark it

...R

That receive with time out works, thankyou so much for this.

Im trying to absorb it and have put comments in the sketch that i'm using,

Can the comments be corrected where i have got them wrong?

Next i will add in the parts that will split the data stream at the commas, are they still called strings then? or tokens when using the strtok method to split and store them.

I want to avoid any use of Strings or other things that cause the arduino to lock up due to running out of memory.

// This sketch uses the Noiasca Liquid Crystal Library, this allows the displaying of German Umlauts using the very common Hitachi HD44780U based LCD's.
// Download the library from https://werner.rothschopf.net/2020/NoiascaLiquidCrystal.zip Simply Extract it to your Arduino libraries folder.
// Some of the characters this library allows you to use are "Ä Ö Ü ä ö ü ß ° µ ÷ · Σ ε ñ £ ¥ π β ~ √ ∝ ∞ ← →"


#include <NoiascaLiquidCrystal.h>      // LCD Library for HD44780U lcd controlers with added Germanu unlauts, an alternative library exists for SPLC780D1 lcd's.
#include <NoiascaHW/lcd_4bit.h>        // Use the parallel interface, 4bit mode. Alternatives are I2C and SPI.

// Set up the LCD:
const byte cols = 24;                 // Columns/characters per row
const byte rows = 2;                  // Rows
const byte rs = 8;                    // Register Select pin
const byte en = 9;                    // Enable pin
const byte d4 = 7;                    // Data bus pin D4
const byte d5 = 6;                    // Data bus pin D5
const byte d6 = 5;                    // Data bus pin D6
const byte d7 = 4;                    // Data bus pin D7
const byte bl = 10;                   // Back light, Set to 255 if not used

const byte numChars = 150;  // Buffer for characters from serial
char receivedChars[numChars]; // Array to store the data from serial port

boolean newData = false;  // Set false at start up?

uint32_t lastReceivedTime;  // Variable to store time last character was received
uint32_t messageTimeout = 50; // Sets time after last character received to  terminate message

LiquidCrystal_4bit lcd(rs, en, d4, d5, d6, d7, bl, cols, rows, convert_special);    //Set up lcd, convert_special = capital and lower case umlauts, i.e Ä gets Ä.

void setup()
{
  lcd.begin();                         // Initialize the LCD
 //lcd.backlight();                     // Turn on backlight
  lcd.createUml();                     // Create 3 German Umlauts using Special Characters 5, 6, and 7 - used for convert_special
    
  pinMode(10, OUTPUT);  // Backlight, via transisstor
  analogWrite(10, 50);  // PWM to backlight, varies brightness

  Serial.begin(115200); // Start serial coms
  lcd.setCursor(0, 0);  // Start at 1st character, top line of lcd
  lcd.print("<Arduino is ready>");  // Writes the phrase to the lcd



}

void loop()
{
  recvWithTimeout();  // Run this item over and over??
  showNewData();  // Ditto?
}

void recvWithTimeout() {
  static byte ndx = 0;  // Stores the incoming bytes from the begining in 'ndx'
  char rc;  // Name the character stream rc?

    if (ndx != 0 && (millis() - lastReceivedTime >= messageTimeout))  // If ndx has something in it, check time and timeout stuff
{
      receivedChars[ndx] = '\0';  // Add the null terminator to end of data stream?
      ndx = 0;  // Set the ndx to zero, ready to receive next stream
      newData = true; // Tells showNewData to do it's stuff
      return; // Exit this part
}    


  while (Serial.available() > 0 && newData == false) {  // Do something only when serial active
    lastReceivedTime = millis();  // Store the time lasst character received?
    
    rc = Serial.read(); // Read the stream on the serial port, store in rc?

      receivedChars [ndx] = rc; // Store the characters?
      ndx++;  // Add the characters coming over serial one after the other
      if (ndx >= numChars) {  
        ndx = numChars -1;  // Remove null terminator?
      }
    }
}



void showNewData() {
  if (newData == true) {  // Do the stuff below when newData has completed
    lcd.setCursor(0, 0);  
    lcd.print("This just in ...       "); // Write to lcd, top line
    lcd.setCursor(0, 1);
    lcd.print(receivedChars); // Write to lcd bottom line
    newData = false;  // Set newData false again, so it starts the program again.
  }
}

Answering your ? in the code

boolean newData = false;  // Set false at start up?

The flag indicates of new data was received on serial. When you start the Arduino, it hasn't received data yet so the flag is set to false.

  recvWithTimeout();  // Run this item over and over??
  showNewData();  // Ditto?

Yes. Serial is slow; you can wait with e.g. a while-loop till you have all data but in that case the processor will not be able to do anything else. E.g. I can imagine that you want to scroll your information over the LCD; if you're stuck in the while-loop(), it can be done but you have to jump through all kinds of hoops to get that going which will result in messy code.

Robin's code works in the following way.

From loop(), you call recvWithXXX followed by showNewData continuously. Starting with showNewData(), it checks the newData flag to determine if a complete message was received and only takes action when that happens. This flag is set in recvWithXXX.

Each time that recvWithXXX is called, it checks if data is available and adds it to the receivedChars buffer till such time that it has determined that the message is complete. When that happens (received the end marker or detected a time out), it sets the flag newData.

  char rc;  // Name the character stream rc?

No, there is no stream in the code. rc probably stands for received character.

    receivedChars[ndx] = '\0';  // Add the null terminator to end of data stream?

receivedChars is an array of characters, not a stream. So you add it to the array at the given position.

    lastReceivedTime = millis();  // Store the time lasst character received?

Correct; maybe better to say "store the time that the last character was received". But that might be a language thing.

    rc = Serial.read(); // Read the stream on the serial port, store in rc?

A little background detail. Serial communication in the Arduino is interrupt driven. Under the hood (invisible to you), there is a 64 byte buffer; each time a new byte is received, it results in an interrupt; a small piece of code (the interrupt service routine) will copy the received byte into that buffer. Serial.read reads from that buffer, not from a stream.

    receivedChars [ndx] = rc; // Store the characters?

Yes, copy the received character to the receivedChars buffer.

    if (ndx >= numChars) {
      ndx = numChars - 1; // Remove null terminator?

No, it does not remove anything. You have created a 150 byte buffer receivedChars. If this snippet wasn't there and you received (and stored) character 151, you would store it in the non-existing position overwriting other variables and possibly causing a crash / undefined behaviour of your code. So this snippet makes sure that you never write outside the boundaries of receivedChars. The last character that you received after 150 will always be stored in the last position of the receivedChars; so if you receive 200 characters, character 200 will be stored in the 150th element. When the code determines that the message is complete, it will overwrite the last received character with the null terminator so you a correct c-string.

Be aware, your receivedChars can hold 150 characters; because of the requirement of the terminating NUL character, you can only receive 149 characters.

1 Like

are they still called strings then

They are strings (lower-case s); in programming guides (for the C language) you will find that word all the time. Nowadays the are often referred to as c-strings (possibly to prevent confusion with String with capital S).

1 Like

Thankyou very much for that,
i now have a slightly better understanding of what's going on, and how the adruino is doing the things i ask of it :slight_smile:

When i made that script, instead of just copying and pasting it i typed out each line, and as i was typing i was trying to 'decode' the words used in my head and think what the processor will be doing with this line of code, which is what i guess programmers do,

Then when i repeat that a few times it will hopefully stick in my head for next time i need to write something in the script to perform a function.

So, what would have been a 2 minute job for most people i imagine, took me most of the day,

But i managed to combine Robin2's ''- Receive with start- and end-markers combined with parsing' and Sterretje's version of 'Robin's receiveWithEndmarker'... 'Receive with timeout'

To make 'Receive with timeout combined with parsing'

I've gone back to using the serial monitor rather than the LCD for this, and i think i need to learn how to use debug commands.

But here is the code i've got so far

// Based on Robin2's example 5... modified from - Receive with start- and end-markers combined with parsing,
// To Receive with timeout (modified by sterretje) combined with parsing, to work with serial data that has no terminator character.

const byte numChars = 32; // Buffer for characters from serial
char receivedChars[numChars]; // Array to store the data from serial port
char tempChars[numChars];       // temporary array for use when parsing

      // variables to hold the parsed data
char messageFromPC[numChars] = {0}; // Text based data
int integerFromPC = 0;  // Numbers
float floatFromPC = 0.0;  // Numbers with points

boolean newData = false;  // Set flag to false on startup, so it can reveive new data

uint32_t lastReceivedTime;  // Variable to store time last character was received
uint32_t messageTimeout = 50; // Sets time after last character received to  terminate message

//============

void setup() {
    Serial.begin(9600); // Starts serial coms
    Serial.println("This demo expects 3 pieces of data - text, an integer and a floating point value");
    Serial.println("Enter data in this style <HelloWorld, 12, 24.7>  ");
    Serial.println(); // Print empty line
}

//============

void loop() {  
    recvWithTimeout();  // Check for data, add it to the receivedChars buffer
    if (newData == true) {  // If 'newData' flag is true, do the following:
        strcpy(tempChars, receivedChars); // Temp copy of receive buffer,  because strtok() used in parseData() replaces the commas with \0
        parseData();  // Split the data into strings?
        showParsedData(); // Place to store the individual strings?
        newData = false;  // Set flag back to false
    }
}

//============

void recvWithTimeout() {
    static byte ndx = 0;  // Stores the incoming bytes from the begining in 'ndx'
    char rc;  // Call the recieved characters 'rc'

    if (ndx != 0 && (millis() - lastReceivedTime >= messageTimeout))  // If ndx has something in it, check time and timeout stuff
      {
    receivedChars[ndx] = '\0'; // Add null terminating character to the end of the array
    ndx = 0;  // Set the ndx to zero, ready to receive next stream
    newData = true; // Tells loop to do it's stuff
  //  return; //Exit this part, dosen't seem to be needed???
  }
          
    while (Serial.available() > 0 && newData == false) {  // Do something only when serial active
    lastReceivedTime = millis();  // Store the time that the last character was received
        rc = Serial.read(); // Read from the serial buffer

                receivedChars[ndx] = rc;  // Copy the characters to the buffer
                ndx++;  // Add the characters coming over serial one after the other
                if (ndx >= numChars) {
                    ndx = numChars - 1; // Allow one less character than the recieveChars buffer, to stop overrun and allow for null terminator
                }
            }

        }
        

//============

void parseData() {      // split the data into its parts

    char * strtokIndx; // this is used by strtok() as an index

    strtokIndx = strtok(tempChars,",");      // get the first part - the string
    strcpy(messageFromPC, strtokIndx); // copy it to messageFromPC
 
    strtokIndx = strtok(NULL, ","); // this continues where the previous call left off
    integerFromPC = atoi(strtokIndx);     // convert this part to an integer

    strtokIndx = strtok(NULL, ",");
    floatFromPC = atof(strtokIndx);     // convert this part to a float

}

//============

void showParsedData() {
    Serial.print("Message ");
    Serial.println(messageFromPC);
    Serial.print("Integer ");
    Serial.println(integerFromPC);
    Serial.print("Float ");
    Serial.println(floatFromPC);
}

Next i want to change it to read multiple strings of plain text instead of: text, number and integer,
But so far havent figured that out.

You mentioned wanting to avoid Strings. Then I would recommend my SafeString lib (V3+) available from Library Manager.
Here is the sketch rewritten using the SafeString lib (see the detailed tutorial here)

// Based on Robin2's example 5... modified from - Receive with start- and end-markers combined with parsing,
// To Receive with timeout (modified by sterretje) combined with parsing, to work with serial data that has no terminator character.
// rewritten by Matthew Ford to use SafeString V3 lib

// install SafeString V3 from Arduino Library manager
#include <SafeStringReader.h>
#include <SafeString.h>
#include <millisDelay.h>

const byte numChars = 32; // Buffer for characters from serial
cSF(sfReceived, numChars); // SafeString for the receive chars
cSF(sfInput, numChars); // SafeString for the input

// variables to hold the parsed data
cSF(messageFromPC, numChars); // Text based data
int integerFromPC = 0;  // Numbers
float floatFromPC = 0.0;  // Numbers with points

uint32_t messageTimeout = 50; // Sets time after last character received to  terminate message

//============

void setup() {
  Serial.begin(9600); // Starts serial coms
  for (int i = 10; i > 0; i--) {
    Serial.print(i); Serial.print(' ');
    delay(500);
  }
  Serial.println();
  SafeString::setOutput(Serial); //uncomment this to enable error msgs
  Serial.println("This demo expects 3 pieces of data - text, an integer and a floating point value");
  Serial.println("Enter data in this style HelloWorld, 12, 24.7  ");
  Serial.println(); // Print empty line
}

//============
bool parseData(SafeString &input);
void showParsedData();

void loop() {
  recvWithTimeout();  // Check for data, add it to the receivedChars buffer
  if (!sfInput.isEmpty()) {  // If have some input, do the following:
    parseData(sfInput);  // Split the data into strings?
    showParsedData(); // Place to store the individual strings?
    sfInput.clear();  // clear processed input
  }
}

//============
millisDelay readTimeOut;

void recvWithTimeout() {
  if (readTimeOut.justFinished()) {
    sfInput = sfReceived;
    sfReceived.clear();
    return;
    // else
  }
  if (sfReceived.read(Serial)) { // read from client into SafeString c, return true if char read
    readTimeOut.start(messageTimeout); // restart time out after each char read
  }
  if (sfReceived.isFull()) { // no more space to read
    SafeString::Output.println("Error: read() filled SafeString sfReceived");
    readTimeOut.stop();
    sfInput = sfReceived;
    sfReceived.clear();
    return;
  }
}

//============

bool parseData(SafeString &input) {      // split the data into its parts
  size_t idx = 0;
  idx = input.stoken(messageFromPC, idx, ',');
  cSF(sfNumber, 20); // to parse numbers
  idx = input.stoken(sfNumber, idx, ',');
  if (sfNumber.toInt(integerFromPC)) {   // valid int integerFromPC is updated
  } else {
    Serial.print(sfNumber); Serial.println(" invalid int");
    return false;
  }
  idx = input.stoken(sfNumber, idx, ',');
  if (sfNumber.toFloat(floatFromPC)) {  // valid float floatFromPC is updated
  } else {
    Serial.print(sfNumber); Serial.println(" invalid float");
    return false;
  }
  return true;
}
//============

void showParsedData() {
  Serial.print("Message ");
  Serial.println(messageFromPC);
  Serial.print("Integer ");
  Serial.println(integerFromPC);
  Serial.print("Float ");
  Serial.println(floatFromPC);
}

For your original question SafeString has a method bool endsWith("gefüllt") which you could use to test the received chars each time one is added, i.e. each time sfReceived.read(Serial) returns true

gazz292:
Next i want to change it to read multiple strings of plain text instead of: text, number and integer,
But so far havent figured that out.

It sounds as if you need to change this line

   integerFromPC = atoi(strtokIndx);     // convert this part to an integer

to another of these

   strcpy(messageFromPC, strtokIndx); // copy it to messageFromPC

with a suitable change to the name of the array where the data should be saved

...R

The SafeString library has another method that will help you with this.
The SafeString.stoken( ) method has an option to return empty fields.
i.e. a,,b will return a b
where as strtok will return only a b That is it skips over all leading delimiters.
This is important for counting fields to get to the one you want.
See the CSV parsing example SafeString_stoken.ino included with the library.
Using stoken() and endsWith() you should be able to code a solution to the original problem you posed.
Finally the LCD updates can be a bit slow, so you may need to add extra input buffering so you don't miss some input.
Text I/O for the Real World goes into that in detail with examples and statistics on how to decide how much extra buffering is needed.

Thankyou, i did look at the safestrings thing before, i actually have the library downloaded, but most of it went over my head.
plus i was worried a little about other people who'd want to use the sketch when i finish it, might be put off having to download a few new libraries just to program an arduino once.

But i guess as they will have to download the Noiasca LCD library if they want to get the German umlauts displaying correctly on the lcd instead of japanese characters, they can download another library just as easy :slight_smile:

At the moment a lot of people are using sketches that use Strings for other connections to the bus simulator (it has 5 com ports open when in full use, one for with dashboard lights, one for the moving needle type gauges, then this IBIS unit, a ticket printer and interior display)
And as a result, the arduinos usually need resetting every 20 to 30 minutes of use, and that becomes normal operation!

I don't want that to accept that, hence needing to move away from the dreaded Strings which i think is the problem (heap memory corruption, which doesn't take long when you only have a few K of memory to start with)

I shall take another look at how to do it the safestrings way, i'm basically learning from the start anyway,
i had a sketch that was working with Strings method, minus the timeout terminator thing (so it wasn't really working with the data that it will be fed, but it worked with a bodge to manually insert a terminator, but that needs doing every time the simulator is started, involving using the task manager and stuff)

There are other things i need to do with this sketch, one of which is split one string at an @ symbol, split another based on it's length, i.e. putting the first character on the top line of the lcd, and the following characters on the lower line,

then having the issue of empty strings skipping to the next one solved, that looks like safestrings does that which is great.

Here is a sketch that seems to do what you want.
It includes self running test data which allows you to test various possible data errors. It runs on Uno (just). The test data takes up memory.
So suggest you move to a Mega2560 for i) more memory and 2) more Serial inputs so you can see your debugging.
The test data is running at 9600, You can try and increase the test baud rate. Again Text I/O for the Real World goes into the details

// install the SafeString library from Arduino library manager
//  see the tutorial at www.forward.com.au/pfod/ArduinoProgramming/SafeString/index.html
// also see Text I/O for the Real World https://www.forward.com.au/pfod/ArduinoProgramming/Serial_IO/index.html#SafeStringStream
#include <SafeString.h>
#include <BufferedOutput.h>
#include <SafeStringStream.h>



/*  Text to use with serial monitor to test , a typical output from the 'IBIS' com port with everything set up on the IBIS:
    1,76,1,0,14:36,- 4.2,E.-dorf Krkhaus     @076   01 105    6 A ,6,105,0,Krankenhaus,noch nicht mit leben gefüllt,

    Output when bus electricity is off, and hence IBIS display is blank / off:
    1,0,0,0,12:45,.,,0,0,-1,,noch nicht mit leben gefüllt,

    I plan on checking for empty 'strings' later, as an empty 'delay' string currently prints weird characters on the LCD.
    And turning the bus electric off leaves the last text on the LCD instead of clearing it */

//#include <LiquidCrystal.h> // use lcd library

//LiquidCrystal lcd(12, 11, 5, 4, 3, 2); // lcd pins

#define TEST_DATA
#ifdef TEST_DATA
const uint32_t TESTING_BAUD_RATE = 9600; // how fast to release the data from sfStream to be read by this sketch
cSF(sfTestData, 340); // the test data SafeString, will be filled in setup()
cSF(rxBuf, 64); // the extra rxbuffer for the SafeStringStream to mimic the Uno hardware serial rx buffer
SafeStringStream sfStream(sfTestData, rxBuf); // set the SafeString to be read from and the SafeString providing the rx buffer

Stream& inputStream = sfStream; // set input to test data;
#else
Stream& inputStream = Serial; // set input to test data;
#endif

cSF(sfReceived, 150); // max looks like 120
createBufferedOutput(bufferedOut, 100, DROP_UNTIL_EMPTY); // create an extra output buffer so debug msgs dont cause missed input

cSF(busStopLine1, 30);
cSF(busStopLine2, 30);
cSF(stopDelay, 10); // for the stop delay

// Set up string index, comments are What the individual strings mean
//field1; // '1' Delay yes / no
//field2; // '76' Bus line number - 3 digits
//field3; // '1' Route
//field4; // '0' Bus stop index
//field5; // '14:36' Time
//field6; // '- 4.2' Delay in minutes and 10ths of a minute, '-xx.x' is early, '+xx.x' is late, '  0.0' is on time
//field7; // 'E.-dorf Krkhaus     @076   01 105    6 A ' Bus stop name, i have changed this with a bodge in Komsi to output 'IBIS lcd contents' 2 lines seperated by @
//field8; // '6' Zone
//field9; // '105' Destination
//field10; // '0' Direction (0 = A, 1 = B, -1 = not set)
//field11; // 'Krankenhaus' Line number - upto 5 digits, another bodge in komsi makes this 'interior display text'
//field12; // 'noch nicht mit leben gefüllt' Interior temp, not working so it displays that text, translated from German =  'not yet filled with life'


void setup() {
  Serial.begin(115200); // start serial coms at the speed indicated
  for (int i = 10; i > 0; i--) {
    Serial.print(' '); Serial.print(i);
    delay(500);
  }
  Serial.println();
  //  lcd.begin(24, 2); // start using lcd, set lcd paramiters, number of colums, rows.

  SafeString::setOutput(Serial); // enable error msgs
  bufferedOut.connect(Serial);
#ifdef TEST_DATA
  Serial.print("Automated Serial testing at "); Serial.println(TESTING_BAUD_RATE);
  sfTestData = F(
                 "1,1,0,14:36,- 4.2,E.-dorf Krkhaus     @076   01 105    6 A ,6,105,0,Krankenhaus,noch nicht mit leben gefüllt," // missing field
                 //                 "1,76,1,0,14:36,- 4.2,E.-dorf Krkhaus     @076   01 105    6 A ,6,105,0,Krankenhaus,noch nicht mit leben gefüllt,"
                 "1,0,0,0,12:45,.,,0,0,-1,,noch nicht mit leben gefüllt,"
                 "1,76,1,0,14:36,- 4.2,E.-dorf Krkhaus     @076   01 105    6 A ,6,105,0,Krankenhaus,noch nicht mit leben gefüllt," // good data
                 "1,0,0,0,12:45,.,,0,0,-1,,noch nicht mit leben gefüllt,"
               ); // initialized the test data
  Serial.println("Test Data:-");
  Serial.println(sfTestData);
  Serial.println();
  sfStream.begin(sfTestData, TESTING_BAUD_RATE); // start releasing sfTestData at 9600 baud call this last
#endif
  // else read from Serial
}

bool processInput(SafeString &input) {
  // do error checking here on input
  // count the number of fields
  size_t idx = 0;
  size_t fieldCount = 0;
  while (idx < input.length()) {
    idx = input.indexOf(',', idx);
    if (idx < input.length()) { // found one
      fieldCount++;
      idx++; // stop over ,
    }
  }
  if (fieldCount != 12) {
    bufferedOut.print(F("Bad data field count:")); bufferedOut.println(fieldCount);
    return false;
  }
  cSF(sfField, 10); // for the fields  does not matter is its too small idx is still updated and empty SafeString returned
  bool returnEmptyFields = true;
  idx = 0;
  for (int i = 1; i < 6; i++)  {
    // skip the first 5
    idx = input.stoken(sfField, idx, ',', returnEmptyFields);
  }
  // pick up delay
  cSF(sfDelay, 10);
  idx = input.stoken(sfDelay, idx, ',', returnEmptyFields);
  // pick up route
  cSF(bussStopName, 100); // for the name
  idx = input.stoken(bussStopName, idx, ',', returnEmptyFields);
  if (bussStopName.isEmpty()) {
    bufferedOut.println(F("Power Off"));
    return false;
  }
  // do field error checking here
  // do not update globals if there are errors
  // if all OK update data
  idx = 0;
  // stoken always clears the return first
  idx = bussStopName.stoken(busStopLine1, idx, '@');
  idx = bussStopName.stoken(busStopLine2, idx, '@');
  stopDelay.clear();
  stopDelay = sfDelay;
  return true;
}

bool getAndProcessInput() {
  bool rtn = false;
  if (sfReceived.read(inputStream)) { // read from input
    if (sfReceived.endsWith("gefüllt,")) {
      // found one process it
      rtn = processInput(sfReceived);
      sfReceived.clear(); // free up for next line
      return rtn;
    }
  }
  if (sfReceived.isFull()) {
    // just keep the last few chars so we will find "gefüllt,"
    bufferedOut.println("sfReceived overflowed, too small or bad data");
    sfReceived.keepLast(10);
  }
  return false; // no new input or have errors
}

void LCDprint() {
  //  // start printing text to the LCD
  //    lcd.setCursor(0, 0); // Sets cursor on the LCD, column, row.
  //    lcd.print(busStopLine1); // Displays the first line of text
  //    lcd.setCursor(0, 1);
  //    lcd.print(busStopLine2); // Displays last line of text
  //    lcd.setCursor(21, 0); // move cursor here, print the delay  -, or + symbol above the numbers as on the real IBIS.
  //   try
  //    lcd.print(stopDelay); //instead of
  /********
    //    lcd.print(stopDelay.charAt(0)); // print the +, -, or nothing, for running late, running early, on time.
    //    lcd.setCursor(20, 1); //bottom row, 21st column
    //    lcd.print(stopDelay.charAt(1)); // first number of delay time, 10's of minutes
    //    lcd.setCursor(21, 1);
    //    lcd.print(stopDelay.charAt(2)); // 2nd number, 1's of minutes
    //    lcd.setCursor(22, 1);
    //    lcd.print(stopDelay.charAt(3)); // 3rd number.. actually a dot.
    //    lcd.setCursor(23, 1);
    //    lcd.print(stopDelay.charAt(4)); // final number, tenths of a minute, i.e. 0 to 60 seconds but shown with with numbers 0 to 9... metric time?!
    //    // i know i could do the above an easier way, something about read everything from the 1st character to the end of that string?
  *******/
}

void loop() {
  bufferedOut.nextByteOut(); // push buffered output to Serial <<<<<<< important

  if (getAndProcessInput()) {  // new data
    bufferedOut.println(busStopLine1);
    bufferedOut.println(busStopLine2);
    bufferedOut.print(" Variation:"); bufferedOut.println(stopDelay);
    LCDprint();
  }
}

drmpf:
Here is a sketch that seems to do what you want.

I was very interested to see a practical example of the use of SafeString. However that code is made very confusing (IMHO) by the streaming facility. I would be interested to see a version of the program that just works on the normal receipt of data via Serial

...R

Comment out this line
#define TEST_DATA
for normal Serial input