Too much NMEA data from a GPS.

Hi,

I'm working with this GPS module .....https://www.sparkfun.com/products/8975. Periodically the GPS unit generates a burst of data (several hundred bytes) that makes up half a dozen NMEA sentences.

My project has two main activities a) it reads and processes GPS data, and b) reads and processes pressure data from a BMP180 module. The project continuously loops and alternates between processing the barograph and reading the GPS. My problem is this, processing the pressure data takes about 35ms, if any GPS data arrives whilst whilst this is happening, there's a good chance it will flood the arduino buffer (64 bytes) and possibly overflow it before the process has a chance of reading all of it. What I'd really like to be able to do is modify the GPS unit so that instead of producing a single data-burst consisting of 6 NMEA sentences, it produces 6 separate burst (one sentence each) separated by several milliseconds, reducing the likelihood of overflowing the input buffer before the data can be read. Does anyone know if this is possible - I've read the datasheet, but there's no obvious clues.

I've also tried to shorten the time take to read the barograph, I've got this down from about 43ms, but that's still too long.

Any suggestions of what else I could try?

Thanks

Do you need all the NMEA sentences that are sent and how often do you need them to be updated?
You can, presumably, configure which sentences the GPS sends and how often it sends them by using the documentation linked to on the Sparkfun page: https://www.sparkfun.com/datasheets/GPS/Modules/PMTK_Protocol.pdf
See the description of the $PMTK314 message starting on page 5.

Pete

Do you need all the NMEA sentences

I don't need all the data, but I do need some data from each sentence. Also, remember, I need to have an entire sentence, before I can check the validation code and confirm that the data is good.

how often do you need them to be updated

I could slow down the update frequency, but I would prefer to have the data refreshed as often as possible.

I'm currently looking at ways of predicting when the next 'burst' of data is likely to start, with the intention of ensuring that none of the other time-consuming processes are also running at that moment.

Take a look at the Adafruit GPS library. Ladyada inherits the serial class and then implemented a double-buffer strategy.

You may also want to look into an RTOS to manage the 35mS pressure sensor.
http://forum.arduino.cc/index.php?topic=144715.0

BigusDickus:

Do you need all the NMEA sentences

I don't need all the data, but I do need some data from each sentence. Also, remember, I need to have an entire sentence, before I can check the validation code and confirm that the data is good.

The best way to proceed is to decide exactly what data you need, and then examine all the sentences. Choose the sentences that contain the most relevant data, then choose the ones that have just a few of the other data.

There's no real predicting needed. You determine the reporting rate. and though I am not familiar with the module you have, many of them will report as quickly as 10 HZ, meaning 10 times per second. You can slow this down as needed, using the configuration strings. You can also configure which sentences are reported, and the reporting interval for EACH sentence. This reporting interval is in units of the overall reporting interval, so if you have it reporting 500 ms (1/2 second), you can configure each sentence's reporting interval in 500 ms increments.

So if there's a sentence that contains a field you need and changes quickly, you set the reporting interval to 1 (and at 2 Hz, that will mean you get that sentence every 500 ms). If another sentence contains slowly changing data, you can configure its report to say, 10, meaing that you get that sentence every 5 seconds.

So if there's a sentence that contains a field you need and changes quickly, you set the reporting interval to 1 (and at 2 Hz, that will mean you get that sentence every 500 ms). If another sentence contains slowly changing data, you can configure its report to say, 10, meaing that you get that sentence every 5 seconds.

Ahhh! - you're saying I can set different sentences to have different frequencies - I didn't realize this, - I will play with this concept and see what I can do.

then implemented a double-buffer strategy.

I've had a quick look at the Adafruit GPS library, but I'm not sure how double buffering might help. Double buffering still requires me to read data from the serial port before it overflows, if another part of my project is running when serial data arrives from the GPS, then I'm still likely to drop data aren't I?

Cheers

Thanks

One option would be to reduce the speed of the serial link so that the burst takes longer to arrive.

Another option would be to hack the serial port interface to increase the receive buffer size.

I don't know what is involved in reading your barometer - does that need to be a blocking activity? Would it cause any harm if you interrupted it?

Or you could get out paper and pen and work out if you can read fill the 64 byte buffer.

Exampl,

At 9600 baud (the default for the GPS) you get (approx) 1000 bytes/chars per second. That's 100 in a 100 ms or 50 in 50 ms and your taking how look playing with the bar graph?

Mark

...then implemented a double-buffer strategy.

I've had a quick look at the Adafruit GPS library, but I'm not sure how double buffering might help. Double buffering still requires me to read data from the serial port before it overflows, if another part of my project is running when serial data arrives from the GPS, then I'm still likely to drop data aren't I?
Cheers
Thanks

The relevant section in Adafruit GPS lib for the double-buffer:

#include <Adafruit_GPS.h>

// how long are max NMEA lines to parse?
#define MAXLINELENGTH 120
// we double buffer: read one line in and leave one for the main program
volatile char line1[MAXLINELENGTH];
volatile char line2[MAXLINELENGTH];
// our index into filling the current line
volatile uint8_t lineidx=0;
// pointers to the double buffers

User code must facilitate the character-by-character transfer of the incoming serial characters into the parsing engine, which is in the background.

void loop() {
  if (Serial.available()) {
    char c = Serial.read();
    Serial1.write(c);
  }
  if (Serial1.available()) {
    char c = Serial1.read();
    Serial.write(c);
  }
}

Code ref: https://github.com/adafruit/Adafruit-GPS-Library/blob/master/examples/flora_gpstest/flora_gpstest.ino

Another approach is to use an interrupt timer to fill the buffer. In this case, loop() is completely free for other activities.

******************************************************************/
// Interrupt is called once a millisecond, looks for any new GPS data, and stores it
SIGNAL(TIMER0_COMPA_vect) {
  char c = GPS.read();
  // if you want to debug, this is a good time to do it!
  if (GPSECHO)
    if (c) Serial.print(c);  
}

void useInterrupt(boolean v) {
  if (v) {
    // Timer0 is already used for millis() - we'll just interrupt somewhere
    // in the middle and call the "Compare A" function above
    OCR0A = 0xAF;
    TIMSK0 |= _BV(OCIE0A);
    usingInterrupt = true;
  } else {
    // do not call the interrupt function COMPA anymore
    TIMSK0 &= ~_BV(OCIE0A);
    usingInterrupt = false;
  }
}

Code ref: https://github.com/adafruit/Adafruit-GPS-Library/blob/master/examples/flora_gpslog/flora_gpslog.ino

Within the Adafruit_GPS.cpp file, you can make changes to the specific sentences parsed. Ex:

 // look for a few common sentences
  if (strstr(nmea, "$GPGGA")) {

Ray

At 9600 baud

The default is 57600, so (in theory) 57 bytes could arrive in 1ms, and over 100 characters could arrive within 2ms. The buffer size is 64 bytes, so the buffer could overflow in less than 2ms.

I've been investigating the barograph module, it take about 34ms to process pressure because after requesting pressure information the code needs to wait for 25ms for the module to respond. I hope to re-write the library so that the 25ms waiting period can be more usefully employed.

Cheers

I've been investigating the barograph module, it take about 34ms to process pressure because after requesting pressure information the code needs to wait for 25ms for the module to respond. I hope to re-write the library so that the 25ms waiting period can be more usefully employed.

Another thought is to separate the LCD/Barometer uC from the GPS uC; that is, use two 328P. You can poll the 328P/GPS to send a data structure of all GPS info in a stream that is pre-parsed. The LCD (slow) and the barometer should be rather happy running together.
http://forum.arduino.cc/index.php?topic=45282.0

http://forum.arduino.cc/index.php/topic,198782.0.html

Refactoring libraries, tweaking, code,and such are great efforts to work out timing/resource issues, but sometimes throwing a $3 atmega328p or a ProMini at the problem is quicker and provides for more flexibility. I especially do this for SD-card logging and IR.

Ray

BigusDickus:
I hope to re-write the library so that the 25ms waiting period can be more usefully employed.

That seems like by far the best approach.

BigusDickus:
Ahhh! - you're saying I can set different sentences to have different frequencies - I didn't realize this, - I will play with this concept and see what I can do.

Here's a little sketch that allows you to send configuration packets to a GPS via Arduino.
The reference at http://api.ning.com/files/L30xp1HTxtEUhhmBj4yYWIB31IDKs*xJNecJYvcEKZofuHJcZ3wnTMuZL5FeJC535I6DJbBZZE7FpJwnQPxgT9yKLCibZaZj/NMEA_Packet_Userl.pdf is quite good. This code allows you to easily send a config sentence, without you having to generate the checksum. You just type in a packet from $ to *, inclusive.

For the frequency of individual packet types, check out $PMTK314.

This code runs on an Arduino Mega2650, but you can run it on an Uno by using Software Serial (or preferably, NewSoftwareSerial) library.

One of these days, if I get time and feel ambitious, I want to write a PC program that will have a good GUI, and allow setting configuration easily.

/*
Enter a packet starting with $PMTK, and ending with a *
In this sketch the * indicates the end of transmission.
The sketch takes the packet, adds a checksum and sends it to
the GPS.

Note: If you change the baud rate of the GPS, you will have to
close the Serial monitor, change the baud rate in Serial1
(or whatever Serial port you use) and restart it.

Set your Serial Monitor to No Line Endings. The sketch supplies those too,
and if you send them, they will get you out of sync.

This sketch assumes you are running an Arduino Mega2560, and
have a serial GPS on Serial1. Change this to suit your own
configuration. (Software Serial for Uno, etc.)

One packet type not mentioned in the document at
http://api.ning.com/files/L30xp1HTxtEUhhmBj4yYWIB31IDKs*xJNecJYvcEKZofuHJcZ3wnTMuZL5FeJC535I6DJbBZZE7FpJwnQPxgT9yKLCibZaZj/NMEA_Packet_Userl.pdf
is packet type 220. Here are a couple of samples, as you
would enter them into the Serial Monitor

$PMTK220,200*
$PMTK220,500*
$PMTK220,1000*
$PMTK220,2000*

These will set the reporting interval to 200, 500, 1000,
and 2000 milliseconds, or 5, 2, 1, .5 Hz,  respectively.
*/

char sentence[80];
int idx = 0;

void setup() {
  // initialize both serial ports:
  Serial.begin(57600);
  Serial1.begin(9600);
}

void loop() {
  // read from Serial1, send to Serial:
  if (Serial1.available()) {
    Serial.write(Serial1.read());
  }

  if (Serial.available())  {
    char inbyte = Serial.read();
    sentence[idx++] = inbyte;
    sentence[idx] = 0;
    if (inbyte == '*') {
      for (int i=0; i < strlen(sentence); i++) {
        Serial.print(sentence[i]);
      }
      Serial.println();
      sendConfig ();
      sentence[0] = 0;
      idx = 0;
    }
  }
}

void sendConfig () {
  unsigned char CS = 0;
  for (int i=1; i< strlen(sentence)-1; i++) {
    CS ^= sentence[i];
  }
  Serial1.print(sentence);
  if (CS < 10) Serial1.print("0");
  Serial1.print(CS,HEX);
  Serial1.println();
}

separate the LCD/Barometer uC from the GPS uC; that is, use two 328P.

I was considering doing this - the final project requires three different Pressure modules - all of which take 34ms to read the pressure. Giving each barograph it's own Arduino Pro-Mini, and then connecting them each to one of a Megas four serial ports may be a viable solution.

Thanks

lar3ry:
Here's a little sketch that allows you to send configuration packets to a GPS via Arduino. This code runs on an Arduino Mega2650...

Thanks for that sketch, just what I needed!

lar3ry:
One of these days, if I get time and feel ambitious, I want to write a PC program that will have a good GUI, and allow setting configuration easily.

I assume you're aware of the existing MiniGPS GUI mentioned at bottom of this Sparkfun GPS module page: https://www.sparkfun.com/products/8975 ?

It doesn't do the rarer commands like $GPRMB, but that's where your sketch comes to the rescue.