Serial.readStringUntil lose characters if insert string fast

DimitrisTzam · February 9, 2021, 1:45pm

Hello all,
Wish a good day to all. Currently I face a problem with Serial.readStringUntil that I can't solve. I will attach my code below. I insert a string via serial monitor. Everything works fine but If I send fast strings via serial monitor then sometimes Serial.readStringUntil doesn't catch right the string. Example string is: $GPRMC,144326.00,A,5107.0017737,N,11402.3291611,W,0.080,323.3,210307,0.0,E,A*20^ with terminator character ^. As you can see from screenshot it didn't read right the string.
I wish I explained the problem right.

#include <string.h>
#include <Wire.h>
#include <Arduino.h>



String ReadString; //Serial string
char ReadCharArray[90];  // characters number
char str2[15];
const char *Wanted_Signals[4] = {"$GPRMC", "$GPGGA", "$GPGSA", "$WIMWV"};
const byte numChars = 85;
char receivedChars[numChars];
boolean newData = false;




void setup() {
  // put your setup code here, to run once:
  Serial.begin(57600);
  Serial.setTimeout(200); // sets the maximum milliseconds to wait for serial data. It defaults to 1000 milliseconds.

}
int i = 0;
void loop() {
  //Serial.flush();
  
  unsigned long StartTime = millis();
  //delay(100);
  //recvWithEndMarker();
  //showNewData();


  //Serial.readBytesUntil('^',ReadCharArray,90);
  ReadString=Serial.readStringUntil('^'); // 94 NMEA data ends with 'return' character, which is ascii(13)
  ReadString.trim();                      // they say NMEA data starts with "$", but the Arduino doesn't think so.
  ReadString.replace(" ","");
  
  Serial.print("String: "); Serial.print(ReadString); Serial.print("\n");
  
 
  
   
  //     $GPGGA,095000.099,3935.008,N,02123.742,E,1,12,1.0,0.0,M,0.0,M,,*65^ 
  //     $GPRMC,144326.00,A,5107.0017737,N,11402.3291611,W,0.080,323.3,210307,0.0,E,A*20^
  //     $GPGSA,M,3,17,02,30,04,05,10,09,06,31,12,,,1.2,0.8,0.9*35^
 // ReadString = "$GPRMC,144326.00,A,5107.0017737,N,11402.3291611,W,0.080,323.3,210307,0.0,E,A*20";
 
  
  ReadString.toCharArray(ReadCharArray,90);
  //Serial.print(ReadCharArray[13]);

  // ######### only for test  ########
  strcpy(str2, Wanted_Signals[1]); 

  //###### $GPRMC
            
  
       if ((!strncmp(Wanted_Signals[0], ReadCharArray, strlen(Wanted_Signals[0]))))
        {
            
*/
        Serial.print("##################");Serial.print("\n");
        }
}

groundFungus · February 9, 2021, 1:53pm

The serial receive buffer is 64 characters by default. If you send more characters, at once, than 64 you are likely to lose some.

The answer is to read the characters as they come in. That way the buffer can't fill. The serial input basics tutorial shows how. A bonus is that the tutorial reads into a null terminated character array, avoiding the problems that the String class can cause (see evils of Strings) and in a non blocking fashion (readString() blocks).

You do need to specify a large enough read buffer in the code to hold the end result (const byte numChars = xx).

DimitrisTzam · February 9, 2021, 1:59pm

Thanks for your reply I appreciate it. So if I increase the buffer it will be ok? The max length of the string will be 85 characters. Also I can't understand why It can read it well but some times no if it's too quick.

drmpf · February 9, 2021, 3:18pm

See my detailed tutorial, Text I/O for the Real World for a reliable fast reader of GPS see my tutorial example which does exactly what you need including filtering msgs and parsing

That tutorial also covers how to increase the input buffer size to larger then 64 bytes and how to see what is going on.

That tutorial also shows you how to add automated test inputs to test how your code handles high speed Serial input using SafeStringStream. Again there is a detailed example in the tutorial

Also the GPS data is terminated by '\n' not '^'. You may have changed this for testing, but SafeStringStream automated testing allow you to use '\n' in the test data and for manual inputs you need only set the Arduino monitor to the Newline setting

DimitrisTzam · February 10, 2021, 9:07am

I followed your advice groundFungus with Example 2 - Receiving several characters from the Serial Monitor and seems to work nice. The problem is when I add an if statement and try to parse the data. If I want to print all these serial prints it passes the if statement and never stops to print also strtok_f() function works for the first time. Now if I only let 3-4 serial.prints if become true and prints one time but if I add string again if stays always false. That’s my code:

// Example 2 - Receive with an end-marker

const byte numChars = 85;
char receivedChars[numChars];   // an array to store the received data

boolean newData = false;

const char *Wanted_Signals[7] = {"$GPRMC", "$GPGGA", "$GPGSA", "$WIMWV", "$SDMTW", "$SDDPT", "$SDDBT"};

void setup() {
    Serial.begin(57600);
    Serial.println("<Arduino is ready>");
}

void loop() {
    recvWithEndMarker();
    showNewData();

   // int checksum = nmea0183_checksum(ReadCharArray,ReadCharArray);  // checksum

     if ((!strncmp(Wanted_Signals[0], receivedChars, strlen(Wanted_Signals[0]))))
        {

             Serial.print("OK");
             Serial.print("match \n");
            char* Message_ID = strtok_f(receivedChars,',');
            char* Time = strtok_f(NULL,',');
            char* Latitude = strtok_f(NULL, ',');
            char* Latitude_direction = strtok_f(NULL,',');
            char* Longitude = strtok_f(NULL,',');
            char* Longitude_direction = strtok_f(NULL,',');
            char* GPS_Quality_Indicators = strtok_f(NULL,',');
            char* Number_of_satellites = strtok_f(NULL,',');
            char* Horizontal_dilution_of_precision = strtok_f(NULL,',');
            char* Antenna_altitude_above = strtok_f(NULL,',');
            char* Units_of_antenna_altitude = strtok_f(NULL,',');
            char* Undulation = strtok_f(NULL,',');
            char* Units_of_undulation = strtok_f(NULL,',');
            char* Age_of_correction_data = strtok_f(NULL,',');
            
           //check = sizeof(str1);
             
            ///check = nmea0183_checksum(str1);
        
            
            Serial.print("The Message ID is : "); Serial.print(Message_ID); Serial.print("\n");
            Serial.print("The Time is : "); Serial.print(Time); Serial.print("\n");
            Serial.print("Latitude is  : "); Serial.print(Latitude); Serial.print("\n");
            Serial.print("Latitude direction is  : "); Serial.print(Latitude_direction); Serial.print("\n");
            Serial.print("Longitude  : "); Serial.print(Longitude); Serial.print("\n");
            Serial.print("Longitude direction is : "); Serial.print(Longitude_direction); Serial.print("\n");
            Serial.print("GPS_Quality_Indicators : " ); Serial.print(GPS_Quality_Indicators); Serial.print("\n");
            Serial.print("Number of satellites : " ); Serial.print(Number_of_satellites); Serial.print("\n");
            Serial.print("Horizontal dilution of precision : " ); Serial.print(Horizontal_dilution_of_precision); Serial.print("\n");
            Serial.print("Antenna altitude above : " ); Serial.print(Antenna_altitude_above); Serial.print("\n");
            Serial.print("Units of antenna altitude : " ); Serial.print(Units_of_antenna_altitude); Serial.print("\n");
            Serial.print("Undulation : " ); Serial.print(Undulation); Serial.print("\n");
            Serial.print("Units of undulation : " ); Serial.print(Units_of_undulation); Serial.print("\n");
            Serial.print("Age of correction data :  " ); Serial.print(Age_of_correction_data); Serial.print("\n");
            Serial.print("******************************************************"); Serial.print("\n");

        }
        
}

void recvWithEndMarker() {
    static byte ndx = 0;
    char endMarker = '^';
    char rc;
   
    while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (rc != endMarker) {
            receivedChars[ndx] = rc;
            ndx++;
            if (ndx >= numChars) {
                ndx = numChars - 1;
            }
        }
        else {
            receivedChars[ndx] = '\0'; // terminate the string
            ndx = 0;
            newData = true;
        }
    }
}

void showNewData() {
    if (newData == true) {
        Serial.print("This just in ... ");
        Serial.println(receivedChars);
        newData = false;

        

    }
}







//###################  STRTOK

char *strtok_fr (char *s, char delim, char **save_ptr)
{
    char *tail;
    char c;

    if (s == NULL) {
        s = *save_ptr;
    }
    tail = s;
    if ((c = *tail) == '\0') {
        s = NULL;
    }
    else {
        do {
            if (c == delim) {
                *tail++ = '\0';
                break;
           }
        }while ((c = *++tail) != '\0');
    }
    *save_ptr = tail;
    return s;
}


char *strtok_f (char *s, char delim)
{
    static char *save_ptr;

    return strtok_fr (s, delim, &save_ptr);
}

DimitrisTzam · February 10, 2021, 9:33am

Also after the first input in the following strings it adds a non visible character in the first place of the array

sterretje · February 10, 2021, 9:57am

numChars should be able to hold the terminating nul character ('\0'); so if the readable length is 85, you need 86. Unless you run low on memory (according to the IDE), add some fat (e.g. make it 100).

Robin2 · February 10, 2021, 10:03am

The strtok() function alters the string it is working on. If you want the original you need to make a copy and apply strtok() to the copy. See the parse example in Serial Input Basics

...R

system · June 10, 2021, 10:04am

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.