Can you do this without changing the SoftwareSerial buffer size?

Using Arduino Networking, Protocols, and Devices
1

Hi,

I am working on a device that, among other functions, has GPS data incoming from a ATGM336H module using the SoftwareSerial library, at a rate of 1Hz
The only data sent to the MPU (atmega4809/nano-every) are only the "$GNRMC" messages (the rest is turned off) in the following format:

$GNRMC,192418.000,A,XXXX.XXXX2,N,00XXX.XXXX1,E,0.00,234.38,290621,011.3,E*62

I am currently using the following code to catch this messages. This is not my entire program, as that is very big, but it's just the relevant parts:

#include <SoftwareSerial.h>
SoftwareSerial GPSModule(10,11); //RX.TX

unsigned long prevGPS;

void setup() {
  Serial.begin(115200);
  GPSModule.begin(9600);
  delay(500);
  Serial.println("start");
  GPSModule.setTimeout(10);
}

void loop() {
  if( millis() - prevGPS >= 1000){
    prevGPS = millis();
    Serial.println(prevGPS);
    if (GPSModule.find("GNRMC,")){
      String GPSmsg = GPSModule.readStringUntil('\n');
      Serial.println(GPSmsg);
    }
  }
}

This displays messages such as

192418.000,A,XXXX.XXXX2,N,00XXX.XXXX1,E,0.00,234.38,2906

which is not the entire message: it's the first 64 characters, from which the "$GNRMC," has been removed (as expected). So the date was cut off, and the magnetic variation and the checksum were removed. The magnetic variation I don't particularly care about, but I do need a checksum.

Now these 64 characters are of course the characters that were saved to the buffer, the rest was not saved.
A solution to this would be to increase the buffer of SoftwareSerial, which apparently is quite easy to do. But this gets advised against over the forum&web, as it increases RAM usage.

the actual question(s):
Now I am wondering if there is another way to do this? Something that comes to mind is continuously monitoring the buffer and removing any characters as soon as they come in, so that you can save them somewhere else and have enough space in the buffer. However I am not sure if you can do that fast enough using an Arduino? The entire message gets send by the GPS-unit as one serial messages, at 9600 baud-rate.
Also, continuously monitoring the buffer would be problematic, as other functions in the device need to be executed much more frequently than this one, some (short ones) at 40Hz.

Greetings,

Chris

2

Deal with each character as it becomes available() and move it to your own buffer. When the message is complete parse your buffer

I suggest that you do not use Strings and that you use a zero terminated array of chars to as your buffer. You can then parse it using the strtok() function

At 9600 baud an Arduino is easily able to deal with this and the suggested method has the advantage that it is non blocking if implemented correctly so you can do other things in between each character if you want

1 Like
3

The serial input basics tutorial shows non-blocking methods to read serial data into a null terminated character array. The data is read into its own buffer that can be any size so the size of the receive buffer is not relevant.

The tutorial includes a parse example using strtok().

1 Like
4

My tutorial Arduino Serial I/O for the Real World covers this buffer problem in detail, including a GPS parsing example.
Your problem may not just be the size of your SoftwareSerial buffer, but that you are printing debug statements that block the loop from reading the SoftwareSerial buffer before it fills up.

Two simple things to do to start with
i) increase your baud rate for Serial, i.e. Serial.begin(115200)
ii) reduce your baud rate for SoftwareSerial, i.e. sf.begin(4800);

If this does not work check the other fixes detailed in the tutorial, including adding your one extra input buffer and timing your loop to see just how slow you are running.

Although I think Strings are often convenient. The Serial find( ) and readStringUntil( ) can be tricky to get running correctly. The Serial.setTimeout( ) that these use defaults to 1sec which in your case is too long.
Try
Serial.setTimeout(50);
but that still may not solve the issues
My tutorial Arduino Software Solutions has various sketches for reading from Serial, with their pros and cons.

If you are worried about RAM usage don't blindly follow the serial input basics tutorial as it allocates char[ ]s the same size as the expected input for every field parsed out. Very safe but in your case will uses a lot of RAM.
Replace its strcpy calls with the safer strlcpy method once you reduce the result char[ ]'s, to keep the code safe from buffer overflows.
This sketch strlcpy_strlcat.ino shows you how to use strlcpy

1 Like
5

My guess is that the problem is you are only looking for input once a second. Try this:

#include <SoftwareSerial.h>
SoftwareSerial GPSModule(10, 11); //RX.TX

void setup()
{
  Serial.begin(115200);
  GPSModule.begin(9600);
  delay(500);
  Serial.println("start");
  // GPSModule.setTimeout(10);
}

void loop()
{
  if (GPSModule.find("GNRMC,"))
  {
    String GPSmsg = GPSModule.readStringUntil('\n');
    Serial.println(GPSmsg);
  }
}

If that works and your loop() does anything else, you will need to detect the arrival of "$GNRMC," without waiting.

1 Like
6

Thanks for the good suggestions so far. I have made the following:

#include <SoftwareSerial.h>
SoftwareSerial GPSModule(10,11); //RX.TX

unsigned long prevGPS;
char messageArray[90] = {0};
bool newMsg = false;

void setup() {
  Serial.begin(115200);
  GPSModule.begin(9600);
  delay(500);
  Serial.println("start");
  GPSModule.setTimeout(10);
}

void loop() {
  if(GPSModule.available() > 0){
    newMsg = bufferRead();
  }
  if( newMsg == true){
    parseFcn();
    newMsg = false;
  }
}

bool bufferRead(){
  bool msgFinished = false;
  while(GPSModule.available() > 0){
    static unsigned int msgPos = 0; 
    char inByte = GPSModule.read();
    if( inByte != '\n' && msgPos < 89){
      messageArray[msgPos] = inByte;
      msgPos++;
    }else{
      messageArray[msgPos] = '\0';
      msgFinished = true;
      msgPos = 0;
      while(GPSModule.available() > 0){
        GPSModule.read();
      }
    }
  }
  return msgFinished;
}

void parseFcn(){
  Serial.println(messageArray);
  static char time_i[5] = {0};
  static char speedOL[6] = {0};
  static char validity = 0;
  
  char *ptr = NULL;
  byte index = 0;
  ptr = strtok(messageArray,",\n");
  while( ptr != NULL){
    switch(index){
      case 1: 
        for(int i = 0; i <= 3; i++){
          time_i[i] = ptr[i];
        }
        time_i[4] = '\0';
      break ;
      case 2:
        validity = ptr[0];
      break;
      case 7:
        for(int i = 0; i <= 4; i++){
          speedOL[i] = ptr[i];
        }
        speedOL[5] = '\0';
      break;
    }
    index++;
    ptr = strtok(NULL, ",");
  }
  Serial.print("time_i = ");
  Serial.print(time_i);
  Serial.print(" , validity = ");
  Serial.print(validity);
  Serial.print(" , speedOL = ");
  Serial.println(speedOL);
}

Which nicely does its job when this is the only code uploaded to the Arduino.
However, when the Arduino is also given other functions to work through, things work for the first couple of messages. But quickly it seems to be missing some characters, and eventually seems to miss the '\n' character, which screws up the entire operation, giving garbage output.

Could this be an indicator of those other functions being blocking? I searched for anything that was blocking, but could not find blocking parts yet. (note that Serial.begin() is actually never called in the actual device, as a separate SPI display is used there.)
Or is there something else that could cause this?

7

Quite likelly

Please post your whole sketch

8

Yes. Like I said:

I would check input characters as they arrive and see if they match or complete the pattern "$GNRMC,".

const char Pattern[] = "$GNRMC,"
byte PatternIndex = 0;

void loop()
{
  if (GPSModule.available())
  {
    chart c = GPSModule.read();
    if (c != Pattern[PatternIndex])
    {
      PatternIndex = 0; // Start over
    }
    else
    {
      // This character matched!
      PatternIndex++;  // Look for the next character
      if (Pattern[PatternIndex] == '\0')
      {
        // Full pattern matched
        String GPSmsg = GPSModule.readStringUntil('\n');
        Serial.println(GPSmsg);
        PatternIndex = 0; // Look for the next match
      }
    }
  }
}
1 Like
9

@chris_abc Nicely coded to avoid char[ ] buffer overflows.

Add a loopTimer to see how slow your loop is running.
Then if you move the loopTimer into bufferRead you can see how slow your sketch is getting back to read the next chars.

Does not clear the rest of the input line only what is currently in the RX buffer.
See How to Flush the Input
If you don't do that you will end up with the next line being a partial (the end of the previous one) which will stuff up you parsing since you are not checking the checksum.

1 Like
10

Again thank you for the replies. I finally had time to look into this better, and have combined some of your suggestions to build this (there is some u8g2 stuff in there that is not directly needed for using the Serial monitor as output, but more on that later.)

#include <SoftwareSerial.h>
SoftwareSerial GPSModule(10,11); //RX.TX

#include <Arduino.h>
#include <U8g2lib.h>

#ifdef U8X8_HAVE_HW_SPI
#include <SPI.h>
#endif

U8G2_ST7565_ERC12864_ALT_F_4W_SW_SPI u8g2(U8G2_R0, /* clock=*/ 2, /* data=*/ 3, /* cs=*/ 4, /* dc=*/ 5, /* reset=*/ 6);

char messageArray[90] = {0};
bool newMessage = false;
bool charLimit = false;
bool timeOut = false;

void setup() {
  Serial.begin(115200);
  GPSModule.begin(9600);
  delay(500);
  Serial.println("start");
}

void loop(){ 
  findReadUntil("$GNRMC",90,'\n',100);  // Continually run the function: (pattern, char limit, terminating char, timeout in ms)
  if(newMessage){
    Serial.println("whole message:");
    Serial.println(messageArray);
    newMessage = false;
  }
  if(charLimit){
    Serial.println("character limit. current messageArray:");
    Serial.println(messageArray);
    charLimit = false;
  }
  if(timeOut){
    Serial.println("timedOut. current smessageArray:");
    Serial.println(messageArray);
    timeOut = false;    
  }
}

void findReadUntil(char pattern[], byte char_lim, char until_c, int timeout_ms){
  static bool searching = true;
  static bool controlledTimeout = false;
  static bool timerRunning = false;
  
  static byte msgIndex = 0;
  static byte patternIndex = 0;
  static unsigned long timerStart = 0;
  
  if( GPSModule.available()>0 ){
  char c = GPSModule.read();
    timerStart = millis();
    timerRunning = true;
    if( searching ){                            // are we searching or writing?
      msgIndex = 0;
      if( c == pattern[patternIndex] ){         // let's see if the character matches the expected pattern
        patternIndex++;                         // if yes, let's continue searching
        if( pattern[patternIndex] == '\0'){
          searching = false;                    // found the end of the pattern, no longer searching
        }
      }else{
        patternIndex = 0;                       // if no, let's start over the search
      }
    }else{                                      // we are writing
      if( msgIndex < char_lim-1 ){              // have we not reached our character limit yet?
        messageArray[msgIndex] = c;             // then let's write
        msgIndex++;
      }else{                                    // if we reached the limit, we give an error and reset
        charLimit = true;
        searching = true;
        controlledTimeout = true;
        timerRunning = false;
        patternIndex = 0;
        msgIndex = 0;
      }
      if( c == until_c){                        // found terminating char? then let's flag a new message, and start over
        newMessage = true;
        searching  = true;
        timerRunning = false;
        patternIndex = 0;
        msgIndex = 0;
      }
    }
  }else{
    if( millis() - timerStart > timeout_ms ){   //over time?
      if( controlledTimeout ){                  // we need to do a controlled timeout if we reached our limit.
        controlledTimeout = false;
        timerRunning = false;
      }else if( timerRunning ){                 // no controlled timeout and timer running? we timed out. 
        timeOut = true;
        searching = true;
        timerRunning = false;
        patternIndex = 0;
        msgIndex = 0;
      }
    }
  }
}

This works well on my separate Arduino when tapping in to the TX stream of the GPS chip. It's also non-blocking, as far as I understand. The output (Serial Monitor) is nicely this:

start
whole message:
,101743.000,A,xxxx.xxxx4,N,0xxxx.xxxx2,E,0.00,79.16,110721,,,A*42

whole message:
,101744.000,A,xxxx.xxxx4,N,0xxxx.xxxx2,E,0.00,79.16,110721,,,A*45

whole message:
,101745.000,A,xxxx.xxxx5,N,0xxxx.xxxx3,E,0.00,79.16,110721,,,A*44

whole message:
,101746.000,A,xxxx.xxxx5,N,0xxxx.xxxx4,E,0.00,79.16,110721,,,A*40

whole message:
,101747.000,A,xxxx.xxxx5,N,0xxxx.xxxx4,E,0.00,79.16,110721,,,A*41

whole message:
,101748.000,A,xxxx.xxxx5,N,0xxxx.xxxx5,E,0.00,79.16,110721,,,A*4F

The device I'm building is not connected directly to computer, but rather runs an LCD screen, using the u8g2 library. Now when I paste the following code in the loop, to run the LCD screen:

if(millis() - prevScreen >= 50){
  prevScreen = millis();
  u8g2.firstPage();
  do {
    u8g2.setDrawColor(1);
    u8g2.setFont(u8g2_font_u8glib_4_tr);
    u8g2.setCursor(100,17);
    u8g2.print("hello");
    u8g2.setCursor(2,30);
    u8g2.print("hello");
    u8g2.setCursor(2,35);
    u8g2.print("hello");
    u8g2.setCursor(20,17);
    u8g2.print("hello");
  } while (u8g2.nextPage() );
}

The sketch remains functional when the message send to the Arduino is short, but when the message becomes longer (e.g. a whole GNRMC NMEA message, ~80 char max, problems start occurring.
The output is now:

start
timedOut. current messageArray:
,102053.000,A,xxxx.xxx54,N,0xxxx.xxxx5,E,0.00,79.16,⸮
whole message:
,102054.000,A,xxxx.xxx57,N,0xxxx.xxxx4,E,0.00,79.16,110721,,,A*46

timedOut. current messageArray:
,102055.000,A,xxxx.xxx59,N,0xxxx.xxxx3,E,0.00,79.16,11072
whole message:
,102056.000,A,xxxx.xxx61,N,0xxxx.xxxx2,E,0.29,79.16,110721,,,A*4C

timedOut. current messageArray:
,102057.000,A,xxxx.xxx70,N,0xxxx.xxxx9,E,0.00,79.16,11072
timedOut. current messageArray:
,102058.000,A,xxxx.xxx77,N,0xxxx.xxxx7,E,0.00,79.16,110721
timedOut. current messageArray:
,102059.000,A,xxxx.xxx84,N,0xxxx.xxxx6,E,0.00,79.16,110721,,,A*
whole message:
,102100.000,A,xxxx.xxx90,N,0xxxx.xxxx4,E,0.00,79.16,110721,,,A*45

Because these problems arise when I actually start using the LCD screen via the u8g2 library, I assume that something about using the u8g2 library in combination with this new non-blocking function is the problem here. Could the u8g2 library maybe be blocking, causing the execution of the function to be too late at times?
What would be a good solution to this?

Also, I am currently using SoftwareSerial, as I did not realize that the Hardware Serial would be much more efficient when designing this thing. Unfortunately I cannot change the current pcb I have made to use the Hardware Serial pins. But if I were to do this in a future design, would this only decrease the load on the cpu, or are there also other advantages to it? (which could potentially solve the issues described above?)

11

HardwareSerial is preferable. Not clear to me if the LCD library is using any interrupts that could interfere with the SoftwareSerial. I suggest you cut some tracks on you pcb and add jumpers to use HardwareSerial

Also you could have careful look at the stepper example in Multi-tasking in Arduino it goes in to detail about giving tasks extra time and how to measure how slow things are being processed.

Your problems may be solved by calling a Serial receive task in between each write to the LCD, see the example code in the above links.

Also read carefully Arduino Serial I/O for the Real World which covers adding an extra RX buffer.

Finally try running your GPS module at 4800 or slower to give the rest of your code more time to run before the RX buffer overflows

12

It got most of your message... I suspect the 100 millisecond timeout is too short. Every 50 milliseconds you take some time to update the screen so you will probably have two screen updates in the message timeout period. Increasing the timeout from 100 to 150 should give you plenty of margin.

13

Thanks for the fast and useful reply!

HardwareSerial is preferable. I suggest you cut some tracks on you pcb and add jumpers to use HardwareSerial

I will definitely use Hardware Serial if I make a new version of this pcb. However, I have ruined some similar pcbs in the past by trying to solder things on/off that were really too small for my soldering skills. This definitely falls into that category, so I want to try to make it work with Software Serial for now, and not run the risk of ruining another pcb.

Finally try running your GPS module at 4800 or slower to give the rest of your code more time to run before the RX buffer overflows

I set the baud rate to 4800 in order to give the MCU more time to keep the buffer clean. Good suggestion, this helped and more of the message is captured now.

I read through the page you posted, and measured the loop timing in different situations. In the following figure, I set out the different measurements for the max loop time, and average loop time, using loopTimer.check(Serial);.
GPS_loop_times
So the maximum loop time is now 65 ms, which seems very long, and is obviously problematic in combination with the screen being updated every 50 ms. I put the screen update rate to 10Hz, but that still gives unstable results.

Your problems may be solved by calling a Serial receive task in between each write to the LCD, see the example code in the above links.

I don't quite understand what you mean here. Could you please elaborate?

Temporary solution
As a temporary solution, I now only allow the screen to only update when I am sure that there is no serial data incoming: The last time at which serial data came in is always saved to a variable lastSerialTime. The screen now only is updated when the following condition holds true:

if( (millis() - prevScreen >= 100) && (millis() - lastSerialTime >= 100) && (millis() - lastSerialTime <= 800) ){
      prevScreen = millis();

So, the screen must wait 100 ms after the last incoming serial data, and has to stop updating ~800 ms after the last serial data came in.

image
image1659×757 41.9 KB

This results in the screen not being able to update for 0.3 seconds, every second. So far I don't find it noticeable, but of course this is not a solution I am completely happy with, hopefully I can solve this better in the future, using Hardware Serial for example (and maybe hardware SPI for the LCD too). For now this will have to make due... (unless your suggestion about a Serial recieve task may help?) In any case, it catches the entire message very reliably.

Also I will test out with higher baud rates how this new solution works out. The screen will not have to wait as long for the serial data to come in, which may give better results.

14

Yes, it's very close to getting it all! I tried your suggestion and went as far as 500 ms. This did not work. I think it is because the timeout for this function controls how long the function will keep trying since the last received character, not since the first received character. So this will unfortunately not help with the buffer overflowing: the MCU simply takes too long at another task, resulting in the buffer filling up, and the buffer not being able to store all of the needed characters. Missing those characters results in the function not being able to find a '\n' within the set time-out, giving the error message.

15

Okay update:
Instead of going this route, I decided to take the more straightforward path of using a larger RX buffer. I simply set it to 90 bytes, as I can easily spare those 26 bytes (ATmega4809 ftw), and it is a much easier, cleaner solution. The same findReadUntil function is still used, which works very well.

16

In principle It goes like this

// some other code here
findReadUntil("$GNRMC",90,'\n',100);
// more code
findReadUntil("$GNRMC",90,'\n',100);
// more code 
findReadUntil("$GNRMC",90,'\n',100);
// etc

That is you call your serial input processing more often. If you move the loopTimer to your findReadUntil("$GNRMC",90,'\n',100); method you can see the delays between it being called.

In practice, your findReadUntil("$GNRMC",90,'\n',100); does some processing which may not allow it to be called multiple times so you set up a RX buffer instead using BufferedInput

createBufferedInput(bufferedIn, 60); // global

in setup()
bufferedIn.connect(GPSModule); // add extra buffering to GPSModule, in addition to the 64byte rx buffer already in SoftSerial

Then use in findReadUntil

if( bufferedIn.available()>0 ){
  char c = bufferedIn.read();

Finally in lot of places in your code to call
bufferedIn.nextByteIn();
To check for chars in SoftwareSerial and transfer them to bufferedIn.

This let you keep the Rx buffer from overflowing by reading chars to the bufferedIn buffer. When you read from bufferedIn it seamlessly takes chars from the buffer and when that is empty takes them from the SoftwareSerial RX buffer.

There are also some stats to help you monitor how full your buffer gets.
see BufferedInput in Arduino Serial I/O for the Real World for the details.

Edit-- If you want to check the delays between calls to bufferedIn.nextByteIn() wrap it in a method and move the loopTimer out of loop() to there (you can create extra named loopTimers if you need to)

void loadBuffer() {
  loopTimer.check(Serial);
  bufferedIn.nextByteIn();
}

and the replace all the calls to bufferedIn.nextByteIn() with loadBuffer(). Now loopTimer will tell you how often bufferedIn.nextByteIn() is being called.

17

This topic was automatically closed 120 days after the last reply. New replies are no longer allowed.