Several OP's have had the problem of not being able to reliably extract various parameters from their GPS/Arduino setup. I think the consensus is that the cause is a timing issue leading to serial buffer overflow.
I want to pin down exactly where and when the overflow occurs so that we can design our code to deliver exactly what we need rather than making stabs in the dark.
I'm thinking of making a set of simple sketches that can be used to demonstrate where & why the overflow occurs. I'll also try to use an oscilliscope to make precise timing measurements.
I hope others can contribute ideas and sketches.
For starters, I think a basic serial pass-thru sketch is an essential tool (you can't really do any meaningful diagnostics until you see the GPS producing intact NMEA sentences):
/*name: ss_passthru_bare_minimum
*
* Bare minimum pass thru sketch.
*If a byte is available on the soft serial it is read and
*sent to the IDE monitor via the hard serial.
*Continues indefinitely.
*
* Try reducing the hard serial baud rate to see if
* the sentences become broken which indicates buffer overflow.
*/
#include <SoftwareSerial.h>
#define pinSoftSerial_RX 3
#define pinSoftSerial_TX 4
SoftwareSerial mySerial(pinSoftSerial_RX, pinSoftSerial_TX);
const int serialBaudRate = 9600, softSerialBaudRate = 9600;
void setup() {
Serial.begin(serialBaudRate);
mySerial.begin(softSerialBaudRate);
}
void loop() {
if (mySerial.available())
Serial.write(mySerial.read());
}
Here is the output at a hard serial baud rate of 9600; all sentences are intact:
and with a hard serial baud rate of 4800; sentences start intact but later ones are broken. So this clearly shows loss of data and it didn't take much to make it happen:
My hardware setup is a u-blox Neo6M GPS receiver on a breakout board labelled GY-NEO6MV2 communicating with a non-genuine Arduino Nano thru software serial. The Nano is connected to the PC/IDE via a USB type B so is via the hardware serial. I will be using the default settings on the neo , namely 9600 baud rate and an output cycle of 1 Hz.
Sorry... but what are we supposed to be seeing?
Oh, okay. I guess I'm over familiar with the NMEA sentences that the GPS outputs.
At each output cycle the GPS sends a bunch of sentences, from 6 to 9. Each starts with "$" and ends with an asterix then two hex digits then CR/LF. So in the first image, you can see the regularity -- "$" at start of each line and "*" and two hex digits at the end. And the CR/LF ensures next data will be on a new line.
Can you see in the second image that the regularity has broken down? That's because bytes have been lost due to buffer overflow. The GPS output cycle starts with the "$GPRMC" sentence (edit: the first GPRMC sentence is at the 9th line, halfway along). The RMC, VTG, GGA, and GSA sentences that follow are intact. The first GSV sentence is not complete and everything from then on is useless until the next output cycle.
Does that help?
Thanks.
What frequency do the messages come in?
and (approx) how many bytes per message are you expecting?
BTW... 4800 baud... approx 480 characters per second.
These are just the things I want to quantify.
The package of messages are produced at 1 Hz. The duration of a package depends on how thoroughly they are populated with data which depends on how many satellites are being "seen". If no sats, it's ~170 characters (bytes) and ~176ms. When they are well populated (about 11 satellites seen there are 480 to 510 bytes, taking ~500ms.
Here's a screenshot example:
Here's where one sat is seen and so the date and time is included (091122 & 101704.00), so more bytes. I've turned on the monitor's time stamp. The "busy_duration" is calculated by the sketch.
And with 11 sats seen, positional fix achieved, so well populated sentences:
Here's the sketch I used to produce the above outputs that give the count and duration:
/*name: ss_passthru__count_bytes
*/
#include <SoftwareSerial.h>
#define pinSoftSerial_RX 3
#define pinSoftSerial_TX 4
SoftwareSerial mySerial(pinSoftSerial_RX, pinSoftSerial_TX);
const int serialBaudRate = 9600, softSerialBaudRate = 9600;
int count;
unsigned long mark1, mark2, busy_duration;
char ch;
bool timeout;
void setup() {
Serial.begin(serialBaudRate);
mySerial.begin(softSerialBaudRate);
}
void loop() {
while (!mySerial.available());
mark1 = millis();
timeout = false;
count = 0;
while (!timeout) {
if (mySerial.available()) {
mark2 = millis();
ch = mySerial.read();
Serial.print(ch);
count = count + 1;
}
else if ((millis() - mark2) > 100) timeout = true;
}
busy_duration = millis() - mark1 - 100;
Serial.print("count:");
Serial.println(count);
Serial.print("busy_duration:");
Serial.print(busy_duration);
Serial.println("ms");
Serial.println();
}
Hello,
Arduino Nano uses 5v pins, the GY-NEO6MV2 Boards I've seen do not include a level converter, does your hardware include this?
To use 4800 baud rate on this device you also have to set the hardware pins, can you confirm the board is setup this way?
The NMEA sentences also include a checksum,
You need to this to verify your data as the UART will not do this automatically
Hi and thanks. What I'm exploring is not a hardware matter. I want to quantify the amount of processing the Nano can do during the GPS output period without causing buffer overflow.
So it's a software matter.
I won't be changing the default configuration of the Neo. That will stay at 9600 and 1Hz. It's the hardware serial baud rate I will be experimenting with. That's between the Nano and the PC/IDE.
I don't think I do -- I think buffer overflow is easy to detect. Once it occurs, the following received bytes are useless. Even the checksum digits can be missing. With no overflow, the sentences are perfect -- I've never had the need to do a checksum check.
The sketch below allows you to select three different outputs:
- bare minimum pass through
- pass through with various counts of bytes processed and durations
- plot of bytes available & received and overflow flag (uses the plotter, not the monitor)
These routines have shown me just how vulnerable the soft serial is to overflow due to time taken to execute code during data receival. I'm puzzled as to how any sketch that uses print statements avoids overflow. I'm yet to see the buffer NOT overflowing when I look at the bytes available. It always seems to hit 62 or 63.
I hope others can make use of this sketch.
/*name: GPS_ss_toolkit
to explore buffer overflow when GPS output
received via software serial
*/
#include <SoftwareSerial.h>
#define LED 13
#define pinSoftSerial_RX 3
#define pinSoftSerial_TX 4
SoftwareSerial mySerial(pinSoftSerial_RX, pinSoftSerial_TX);
const int serialBaudRate = 19200, softSerialBaudRate = 9600;
//case 1: bare minimum, 2: counts & durations
//3: plot bytes available & overflow flag
unsigned long mark1, time_measure1 = 0;
char ch;
String overflowOccurred;
bool timeout;
unsigned long mark2, duration, busyDuration, idleDuration,
timeoutValue = 100;
int case_selected, bytesReceived, charsReceived, bytesAvail, overflowFlag;
void setup() {
Serial.begin(serialBaudRate);
mySerial.begin(softSerialBaudRate);
}
void loop() {
Serial.println("GPS software serial toolkit");
Serial.println("===========================");
Serial.println();
Serial.println("Select from:");
Serial.println();
Serial.println("1: bare minimum pass through");
Serial.println("2: pass through with counts & durations");
Serial.println("3: plot bytes available, overflow flag & bytes received");
Serial.println();
Serial.println("Enter 1 or 2 in the input box above, or");
Serial.println();
Serial.println("for selection 3:");
Serial.println("close the monitor, open the plotter ");
Serial.println("and enter '3' in the input box at bottom");
Serial.println();
while (!Serial.available()) {};
int case_selected = Serial.parseInt();
Serial.println(case_selected);
switch (case_selected) {
case 1:
//bare minimum
//************
Serial.println("case 1");
while (true) {
if (mySerial.available()) {
Serial.write(mySerial.read());
}
}
break;
case 2:
Serial.println("case 2");
//count bytes and measure durations
//*********************************
while (true) {
while (!mySerial.available()) {}; //wait till GPS output starts
timeout = false;
mark1 = millis(); //marks start of GPS busy period
charsReceived = 0;
overflowOccurred = "no";
idleDuration = mark1 - mark2;
//GPS busy period -- echo GPS output to serial
while (!timeout) {
if (mySerial.available()) {
mark2 = millis(); //time when most recent byte received
if (mySerial.overflow()) {
overflowOccurred = "yes";
digitalWrite(LED, HIGH);
ch = mySerial.read(); //read the byte
Serial.print("^"); //print it
}
else {
digitalWrite(LED, LOW);
ch = mySerial.read(); //read the byte
Serial.print(ch); //print it
}
charsReceived = charsReceived + 1; //increment characters read count
}
else {
duration = (millis() - mark2);
if (duration > timeoutValue) timeout = true; //no bytes avail,so busy period over
}
}
//GPS idle period -- report results
busyDuration = mark2 - mark1;
Serial.println();
Serial.print("bytes received: "); Serial.print(charsReceived); Serial.println(" x");
Serial.print("busy duration: "); Serial.print(busyDuration); Serial.println(" ms");
Serial.print("idle duration: "); Serial.print(idleDuration); Serial.println(" ms");
Serial.print("complete cycle: "); Serial.print(idleDuration + busyDuration); Serial.println(" ms");
Serial.print("overflow occurred: "); Serial.print(overflowOccurred);
Serial.println(); Serial.println();
}
break;
case 3:
//plot bytes available & overflow indicator
//*****************************************
//must open plaotter, not monitor
Serial.println("case 3");
while (true) {
bytesAvail = (mySerial.available());
if (mySerial.overflow()) {
digitalWrite(LED, HIGH);
overflowFlag = 50;
}
else {
digitalWrite(LED, LOW);
overflowFlag = 0;
}
if (bytesAvail > 0) {
ch = mySerial.read();
bytesReceived = bytesReceived + 1;
}
else bytesReceived = 0;
Serial.print(bytesAvail);
Serial.print(" ");
Serial.print(overflowFlag);
Serial.print(" ");
Serial.println(bytesReceived);
}
break;
}
}
Use a much higher baud rate. 9600 baud was fast, for the 1980's.
Do less output. Abbreviate.
Do output less frequently. Use a millis() timer to output status once a second or less frequently.
The baud rate on the hardware Serial port (the port going to the USB) is not going to markedly affect the amount of time you have for processing, provided you never fill the transmit buffer. That is the only time the actual baud rate will have an effect, because the code is having to wait for space to become available in the buffer. Running a lower baud rate than the software serial port is just asking for trouble if you want to re-transmit all the data being received.
If you want more processing time, get a board with two hardware serial ports, that relieves the arduino of the burden of running software serial.
Thanks, jw. I'm surprised to find I can't use higher than 19200 on my non-genuine Nano.
Here's some pics of typical outputs from the sketch:
case 1, no overflow at 19200:
case 2, no overflow at 19200:
case 2, overflow at 19200:
case 3, no overflow at 19200:
In the last pic from the plotter, blue trace is bytes available, red is overflow flag (zero: no overflow, 50: overflow) and green is bytes received.
When receiving at 9600 baud / transmitting at 4800 baud, you can only transmit one character in the time it takes to receive two. If you are receiving a constant stream of characters, with no breaks between characters, the transmit buffer will be full after receiving 128 characters (you have had time to actually transmit 64 of those characters, the remainder are filling the tx buffer). After that, the receive buffer starts to fill, since print becomes blocking and you will be receiving two characters in the time it takes to transmit one. Inevitable buffer overflow if the data input does not stop.
Why can't the nano run higher than 19200 baud? Only problem I've had, even with cheap nano copies, is a raspberry pi that could not keep up with receiving something like 250,000 baud.
I don't use 4800 in practice -- it's just a quick n easy way to force overflow for the purposes of testing the sketch.
The real purpose of this "project" is to determine the limits on executing code during the GPS output.
Thanks for the interest.
I don't know why the Nano can't use greater than 19200 -- I just don't get any data appearing in the IDE at higher baud rates.
Your Nano is faulty? That does not bode well for the usefulness of your testing. Time to get a Nano that works at 115200 or 250000 baud, like every other Arduino.
So clearly bytes are being received quite a bit slower than the bytes are entering the buffer. Try adding a trace for available space in the output buffer (Serial.availableForWrite()) to see if the output buffer is filling up.
Thanks, jw, I didn't know about that function. Will try that later on.
As long as the transmit buffer is never full, the transmit baud rate should be irrelevant. The baud rate will not affect how long it takes to place a character in the tx buffer, or how long or how many times the Serial interrupt occurs.
I don't really see how forcing buffer overflow with a slow transmit baud rate helps you determine anything about available processing time. Instead, put a delay (either delay() or delayMicroseconds() ) in the test code, and see how much you can increase that before overflow occurs (running tx at a decently high baud rate).
Thanks for that. So when does higher baud rate help?
It doesn't. As I said it's a simple way of testing the sketch.
Yes, that's what I intend. I've already learned it doesn't take many ms delay to trigger overflow, but I want to quantify it more precisely and relate that to how much code can be executed without causing overflow.
