Serial_buffer overflow at second hardware serial

Projects Programming
1

Hi all,

I'm trying to develop a dual GPS serial logger with continues streams of data using a SAMD21.

A second Serial has been set up in the variants.h using a free sercom.
Both modules can run on various baudrates, using 460800 atm.
Communication to both modules works fine.

I'm having trouble with understanding how the internal serial ringbuffer works for different ports.
The stream of data from the first port is printed fine, while the second overflows at the size of serial_buffer from Ringbuffer.h

Increasing the size of the ringbuffer helps a bit, but still not all data is being received from the second port.

#include <Arduino.h>

#define BASE  Serial1
#define ROVER Serial2
#define GPS_BAUD 460800

#define CONSOLE_STREAM SerialUSB
#define CONSOLE_BAUD 115200

#define READ_TIMEOUT_MS 2

#define BUFFER_SIZE 4096
uint8_t buffer[BUFFER_SIZE];

void setup()
{
  BASE.begin(GPS_BAUD);
  ROVER.begin(GPS_BAUD);
  CONSOLE_STREAM.begin(CONSOLE_BAUD);
}

void loop()
{
  // BASE -> Console
  CONSOLE_STREAM.print("\n\n\rBASE: ");
  memset(buffer, 0, BUFFER_SIZE);
  writeConsole( readSerialStream((Stream*)&BASE) );

  // ROVER -> Console
  CONSOLE_STREAM.print("\n\n\rROVER: ");
  memset(buffer, 0, BUFFER_SIZE);
  writeConsole( readSerialStream((Stream*)&ROVER) );
}


size_t readSerialStream(Stream* stream)
{
  uint32_t last = millis();
  size_t count = 0;
  while ((count < BUFFER_SIZE) && (millis() < (last + READ_TIMEOUT_MS))) {
    if (stream->available()) {
      buffer[count++] = stream->read();
      last = millis();
    }
  }
  CONSOLE_STREAM.println(count);
  return count;
}


void writeConsole(size_t count)
{
  if (count > 0) {
    for (size_t i = 0; i < count; i++) {
      CONSOLE_STREAM.print((char)buffer[i]);
    }
  }
}

Below a single read of both receivers:

  • The data aquired from the BASE module is as expected.
  • The data aquired from the ROVER module is capped at the buffer size. (overflow)
BASE Bytes: 1886 
$GNRMC,175823.00,A,XXXX.XXXXX,N,XXXXX.XXXXX,E,0.023,,200320,,,A,V*16
$GNVTG,,T,,M,0.023,N,0.042,K,A*3A
$GNGGA,175823.00,XXXX.XXXXX,N,XXXXX.XXXXX,E,1,12,0.62,3.6,M,46.0,M,,*42
$GNGSA,A,3,20,27,16,21,26,10,29,,,,,,1.09,0.62,0.89,1*01
$GNGSA,A,3,65,73,82,79,66,81,80,,,,,,1.09,0.62,0.89,2*0D
$GNGSA,A,3,27,01,13,15,21,,,,,,,,1.09,0.62,0.89,3*0F
$GNGSA,A,3,24,13,26,20,29,,,,,,,,1.09,0.62,0.89,4*00
$GPGSV,3,1,12,05,01,020,09,07,05,338,22,08,08,274,16,10,11,159,28,1*60
$GPGSV,3,2,12,16,69,278,35,18,42,059,34,20,33,136,33,21,65,077,37,1*63
$GPGSV,3,3,12,26,65,188,39,27,39,277,28,29,15,087,35,31,07,201,,1*6C
$GPGSV,3,1,12,05,01,020,,07,05,338,,08,08,274,,10,11,159,19,6*6B
$GPGSV,3,2,12,16,69,278,,18,42,059,27,20,33,136,,21,65,077,,6*64
$GPGSV,3,3,12,26,65,188,,27,39,277,,29,15,087,17,31,07,201,,6*6B
$GLGSV,3,1,10,65,20,030,24,66,17,094,32,72,03,352,,73,17,332,21,1*74
$GLGSV,3,2,10,79,30,187,29,80,60,263,22,81,51,048,27,82,66,277,35,1*73
$GLGSV,3,3,10,83,20,250,,88,08,060,25,1*7E
$GLGSV,3,1,10,65,20,030,,66,17,094,,72,03,352,14,73,17,332,,3*77
$GLGSV,3,2,10,79,30,187,,80,60,263,,81,51,048,30,82,66,277,26,3*7E
$GLGSV,3,3,10,83,20,250,,88,08,060,,3*7B
$GAGSV,3,1,09,01,23,316,25,03,04,089,,05,09,041,26,09,02,355,09,7*71
$GAGSV,3,2,09,13,66,231,30,15,57,062,31,21,63,264,30,26,13,236,25,7*79
$GAGSV,3,3,09,27,39,175,34,7*41
$GAGSV,3,1,09,01,23,316,,03,04,089,,05,09,041,,09,02,355,,*4C
$GAGSV,3,2,09,13,66,231,,15,57,062,,21,63,264,,26,13,236,,*4B
$GAGSV,3,3,09,27,39,175,,*71
$GBGSV,3,1,09,02,00,098,,05,15,119,,13,17,048,26,19,04,143,,1*70
$GBGSV,3,2,09,20,22,097,33,24,22,317,22,26,45,264,12,29,61,073,37,1*7B
$GBGSV,3,3,09,30,08,077,22,1*44
$GBGSV,3,1,09,02,00,098,,05,15,119,,13,17,048,,19,04,143,,*45
$GBGSV,3,2,09,20,22,097,,24,22,317,,26,45,264,,29,61,073,,*4D
$GBGSV,3,3,09,30,08,077,,*75
$GNGLL,XXXX.XXXXX,N,XXXXX.XXXXX,E,175823.00,A,A*77

ROVER Bytes: 255
$GNRMC,175823.00,A,XXXX.XXXXX,N,XXXXX.XXXXX,E,0.040,,200320,,,A,V*11
$GNVTG,,T,,M,0.040,N,0.074,K,A*3A
$GNGGA,175823.00,XXXX.XXXXX,N,XXXXX.XXXXX,E,1,12,0.59,17.5,M,46.0,M,,*7E
$GNGSA,A,3,21,27,16,20,26,29,10,,,,,,1.04,0.59,0.86,1*0B
$GNGSA,A,3,66,80,7

After reading the Serial Input Basics tutorials I've changed the readSerialStream function to:

size_t readSerialStream(Stream* stream)
{
  uint32_t last = millis();
  size_t count = 0;
  if (stream->available() > 0)
  {
    while ((stream->available() > 0) && (count < BUFFER_SIZE)) {
      buffer[count++] = stream->read();
    }
  }
  CONSOLE_STREAM.println(count);
  return count;
}

The output of the last function: (increased the serial_buffer_size to 1024)

BASE: 0

ROVER: 41
$GNRMC,225341.00,A,XXXX.XXXXX,N,XXXXX.XXX

BASE: 122
$GNRMC,225341.00,A,XXXX.XXXXX,N,XXXXX.XXXXX,E,0.056,,200320,,,A,V*12
$GNVTG,,T,,M,0.056,N,0.104,K,A*3B
$GNGGA,225341.00,

ROVER: 679
42,E,0.047,,200320,,,A,V*11
$GNVTG,,T,,M,0.047,N,0.087,K,A*31
$GNGGA,225341.00,XXXX.XXXXX,N,XXXXX.XXXXX,E,1,12,0.62,2.9,M,46.0,M,,*49
$GNGSA,A,3,01,03,11,14,28,08,22,32,17,,,,1.07,0.62,0.87,1*0E
$GNGSA,A,3,76,69,84,68,85,,,,,,,,1.07,0.62,0.87,2*0E
$GNGSA,A,3,31,26,33,24,07,08,13,,,,,,1.07,0.62,0.87,3*03
$GNGSA,A,3,09,06,12,16,21,22,,,,,,,1.07,0.62,0.87,4*01
$GPGSV,3,1,12,01,77,300,43,03,38,230,20,08,32,169,31,10,08,058,20,1*68
$GPGSV,3,2,12,11,66,147,42,14,43,097,31,17,18,316,18,22,66,225,35,1*6D
$GPGSV,3,3,12,24,02,005,22,27,05,155,05,28,26,289,29,32,37,065,36,1*67
$GPGSV,3,1,12,01,77,300,36,03,38,230,,08,32,169,30,10,08,058,17,6*6A
$GPGSV,3,2,12,11,66,147,,1

BASE: 1023
XXXX.XXXXX,N,XXXXX.XXXXX,E,1,12,0.60,19.8,M,46.0,M,,*73
$GNGSA,A,3,03,08,11,14,17,28,22,32,01,,,,1.05,0.60,0.86,1*0F
$GNGSA,A,3,76,68,84,83,69,85,,,,,,,1.05,0.60,0.86,2*04
$GNGSA,A,3,33,26,13,31,24,07,08,,,,,,1.05,0.60,0.86,3*02
$GNGSA,A,3,09,12,16,21,19,06,22,,,,,,1.05,0.60,0.86,4*08
$GPGSV,3,1,12,01,77,300,35,03,38,230,18,08,32,169,27,10,08,058,20,1*65
$GPGSV,3,2,12,11,66,147,31,14,43,097,32,17,18,316,25,22,65,225,33,1*61
$GPGSV,3,3,12,24,02,005,15,27,05,155,16,28,26,289,28,32,37,065,23,1*64
$GPGSV,3,1,12,01,77,300,31,03,38,230,16,08,32,169,26,10,08,058,18,6*62
$GPGSV,3,2,12,11,66,147,,14,43,097,,17,18,316,13,22,65,225,,6*60
$GPGSV,3,3,12,24,02,005,,27,05,155,,28,26,289,,32,37,065,29,6*60
$GLGSV,3,1,09,68,40,041,29,69,75,149,36,70,20,203,,75,06,289,19,1*78
$GLGSV,3,2,09,76,15,335,18,77,05,025,,83,18,128,,84,78,104,26,1*70
$GLGSV,3,3,09,85,39,316,28,1*48
$GLGSV,3,1,10,68,40,041,22,69,75,149,34,70,20,203,,75,06,289,,3*73
$GLGSV,3,2,10,76,15,335,20,77,05,025,,83,18,128,20,84,78,104,35,3*71
$GL

ROVER: 1023
4,43,097,,17,18,316,19,22,66,225,,6*69
$GPGSV,3,3,12,24,02,005,,27,05,155,17,28,26,289,,32,37,065,21,6*6E
$GLGSV,3,1,09,68,40,041,31,69,75,149,37,70,20,203,,75,06,289,22,1*78
$GLGSV,3,2,09,76,15,335,,77,05,025,,83,18,128,,84,78,104,20,1*7F
$GLGSV,3,3,09,85,39,316,24,1*44
$GLGSV,3,1,10,68,40,041,28,69,75,149,35,70,20,203,,75,06,289,,3*78
$GLGSV,3,2,10,76,15,335,22,77,05,025,,83,18,128,,84,78,104,35,3*71
$GLGSV,3,3,10,85,39,316,28,3*42
$GAGSV,3,1,10,07,57,103,31,08,24,045,24,12,07,297,19,13,14,118,36,7*76
$GAGSV,3,2,10,14,16,155,23,24,14,320,14,25,00,006,,26,67,120,16,7*76
$GAGSV,3,3,10,31,11,271,13,33,57,297,33,7*78
$GAGSV,3,1,10,07,57,103,,08,24,045,,12,07,297,,13,14,118,,2*7A
$GAGSV,3,2,10,14,16,155,,24,14,320,,25,00,006,,26,67,120,29,2*7B
$GAGSV,3,3,10,31,11,271,,33,57,297,,2*7F
$GBGSV,3,1,09,05,14,120,,06,20,037,25,09,36,059,31,12,75,189,36,1*7F
$GBGSV,3,2,09,16,26,042,31,19,12,048,,21,55,198,27,22,60,073,37,1*7E
$GBGSV,3,3,09,23,00,311,,1*4D
$GBGSV,3,1,09,05,14,120,,06,20,037,,09,36,059,

BASE: 0

ROVER: 0

Still overflowing the ring buffer and thus causing gaps in the data.
What could be an adequate solution to this issue?

2

Any reason why you must use such an incredibly high baud rate? What happens at 115200 baud?

3

Also, do you need all NMEA sentences enabled? Getting rid of all the unnecessary ones will help prevent the overflow. However, you will need to ensure your code is fast enough to handle the remaining NMEA sentence(s).

4 
  size_t count = 0;

Is it possible that a size_t type occupies only a byte in this toolchain? Then asking for a 1024 byte buffer would be in vain. Edit - Oh well, I see it has to be 16 bits... well, then I don't know...

5

You have two serial lines coming in at 460800, and expect to be able to print the combined data to the console at 115200? Remember if you overflow the serial write buffer it will stall the code waiting for space in the send buffer.

6

Thanks for the responses!

The software has a lot more to do than just aquiring this data. With the higher rate, more symbols are transferred over the line each second thus reducing the tx time of the data. Tried it with 115200 and 38400 but no difference.

aarg:
Any reason why you must use such an incredibly high baud rate? What happens at 115200 baud?

The NMEA messages are just to validate the incoming data using it's timestamp. The final result needs raw GNSS observations, which is even more data but more work to extract the timestamp from. A Rpi3 has been used in the past communicating with the modules over USB but the goal is to get it working on an embedded platfrom.
Could you elaborate on ensuring the code is fast enough, how could make it faster as it is now?

Power_Broker:
Also, do you need all NMEA sentences enabled? Getting rid of all the unnecessary ones will help prevent the overflow. However, you will need to ensure your code is fast enough to handle the remaining NMEA sentence(s).

The write buffer doesn't seem the problem. It's the receive buffer. All messages are produced every second so a block in the writing is accepted.

david_2018:
You have two serial lines coming in at 460800, and expect to be able to print the combined data to the console at 115200? Remember if you overflow the serial write buffer it will stall the code waiting for space in the send buffer.

Can someone explain (or has a clue) what might be the cause of only the second buffer being overflowed? From my understanding the buffer is initialized twice whit Serial1.begin and Serial2.begin.

7

All messages are produced every second so a block in the writing is accepted.

No, one second is the interval between data bursts. During a burst, the output will block as soon as the output buffer is full (which will happen almost immediately in your case). While blocked, the input buffer is not being serviced.

Just to be clear, have you set both input streams to less than half the output stream baud rate? I see you did try 38400 but was it on all your inputs?

I understand that it is strange that only one input is having a problem, but the underlying cause may not be so obvious. Have you tried swapping inputs?

8

Thanks for the response @aarg,

It's reading the buffer in a loop byte per byte, and afterwards printing them byte by byte. These are two separate functions how can it be blocking?

aarg:
No, one second is the interval between data bursts. During a burst, the output will block as soon as the output buffer is full (which will happen almost immediately in your case). While blocked, the input buffer is not being serviced.

Yes, both inputs have been set to 38400

aarg:
Just to be clear, have you set both input streams to less than half the output stream baud rate? I see you did try 38400 but was it on all your inputs?

Yes, swapped them to check if it maybe was something else causing the problem. Same results but then for the other module.

aarg:
I understand that it is strange that only one input is having a problem, but the underlying cause may not be so obvious. Have you tried swapping inputs?

9

What about using a larger BUFFER_SIZE or READ_TIMEOUT_MS? Maybe a pause in the input stream is timing it out. Can you add a debug print for a time out event?