20Hz GPS (115200 serial) and serial parsing

[fuzion]

Hello all

I decided to post here as i cannot seem to find a proper solution for my issue.

I am working on a personal project to create a laptimer for tracks with GPS. My hardware is:

Arduino mega 2560
Sparkfun venus gps (20hz capable) with external active 3V antenna
Digole serial display 320x240 I2C interface

So here it is:

The GPS sends data succesfully at 20Hz through serial. Only with a mega, as the softserial of arduino uno had issues. I use a serial passthrough program like this:

Code: [Select]

void setup() {
  // initialize both serial ports:
  Serial.begin(115200);
  Serial1.begin(115200);
}

void loop() {
  // read from port 1, send to port 0:
  if (Serial1.available()) {
    int inByte = Serial1.read();
    Serial.write(inByte);
  }

  // read from port 0, send to port 1:
  if (Serial.available()) {
    int inByte = Serial.read();
    Serial1.write(inByte);
  }
}

to read the serial data and send commands to the GPS module. The module works only at 115200 at that speed, 38400 for 10Hz, 9600 for below.

The problems start when i try to parse the data AND do something else on the program, like millis() processing for laptimes or writing stuff on the I2C display. If i just parse the data with this code:

Code: [Select]

void loop() {

  if (Serial1.available())
  {
    char ch = Serial1.read();
    Serial.write(ch);
    if (ch != '\n')
    {
      sentence[i] = ch;
      i++;
    }
    else
    {
      i = 0;
      nmea++;
    }
  }
}


i can properly see the data on a terminal from serial0 of arduino mega. The data is this per sentence:

$GPRMC,220516,A,5133.82,N,00042.24,W,173.8,231.8,130694,004.2,W*70

With a \n character for new line.

Now:

- If i perform any calculations under the else{} of the serial parsing, and by using something like %20 to display things on the screen with a small delay (sentences come every 50msec,i refresh the screen every 500msec as it is slow) everything is good. but then all my processing is limited in 50ms steps of the GPS speed (which is not that bad for the laptimer as it has 50ms resolution anyway limited from the gps - but its not nice to have)

- If i put my code outside the serial parsing, the terminal then gets very slow or full of errors. I tried timing the rest of the code without delay() (with millis like the blink without delay example), no luck. If i use 1ms delay() on the loop{} void, the serial is messy again (it gets messy even without any more code like this).

It seems i cannot do anything here and i don't understand why. How are simple math calculations messing with the Serial.read() loop? Any ideas welcome.

[Robin2]

Have a look at the examples in Serial Input Basics - simple reliable ways to receive data. There is also a parse example.

What is the purpose of the pass-through arrangement? Is it only for testing?

If this was my project I think I would receive a complete message before passing on the message.

...R

[fuzion]

hey. thanks for the reply.

the pass-through is only for testing yes, and programming of the GPS module with its own software (i couldnt work it with a usb to serial converter so i used the passthrough succesfully). but if the pass through has problems, i get corrupted data also from the Serial1.read in general. I mean even if i comment the passthorugh out on my program (i have a simple serial.write under the serial1.read and a seril.begin(115200) ), the reading is still problematic.

i have read the serial basics but again i dont see anything wrong with my parsing, which works fine unless i do other stuff under or before it. weird. i thought it had something to do with the sentence speed and time space between sentences at 115200 but i dont know.

what do you mean you would read the complete message before the parsing? on my code i create a sentence array until i receive the \n, and then i do stuff with each sentence char.

for example, to get the latitude in decimal degrees from char NMEA format:

Code: [Select]

if (Serial1.available())
  {
    char ch = Serial1.read();
    Serial.write(ch);
    if (ch != '\n')
    {
      sentence[i] = ch;
      i++;
    }
    else
    {
      i = 0;
      nmea++;     
     
      latitude = (byte(sentence[20]) - 48) * 10 + (byte(sentence[21]) - 48) + (float(
                   (long(sentence[22]) - 48) * 100000
                   + (long(sentence[23]) - 48) * 10000
                   + (long(sentence[25])  - 48) * 1000
                   + (long(sentence[26]) - 48) * 100
                   + (long(sentence[27]) - 48 ) * 10
                   + (long(sentence[28]) - 48 )) / float(600000));
     }
  }

[Robin2]

i have read the serial basics but again i dont see anything wrong with my parsing, which works fine unless i do other stuff under or before it. weird. i thought it had something to do with the sentence speed and time space between sentences at 115200 but i dont know.

what do you mean you would read the complete message before the parsing? on my code i create a sentence array until i receive the \n, and then i do stuff with each sentence char.

I don't think you have posted the complete program that you are having trouble with. Without seeing that it is very hard to help.

In the first program you posted you are not receiving the full message - you are passing on each byte as it arrives.

My purpose in suggesting you study my Serial Input Basics was so that you can see how the task is broken out into separate functions. One function to receive the data, another to parse it and another to show the values. That makes it much easier to manage the project.

...R

[fuzion]

what i see on the basics is the example 2 lets say, which again creates a character array sentence until a terminating character, like i do.

where sentence=receivedChars and my \n = endMarker

Code: [Select]

while (Serial.available() > 0 && newData == false) {
        rc = Serial.read();

        if (recvInProgress == true) {
            if (rc != endMarker) {
                receivedChars[ndx] = rc;
                ndx++;
                if (ndx >= numChars) {
                    ndx = numChars - 1;
                }
            }
            else {
                receivedChars[ndx] = '\0'; // terminate the string
                recvInProgress = false;
                ndx = 0;
                newData = true;
            }
        }

i will post my programs tonight, the version that works and the version that doesn't.

[fuzion]

ok did it now. the only thing i change is the bracket of else statement, if i include everything until the end of program it works, if i close the else after the speedkmh it scrambles the data.

Works, serial monitor: mon.jpg:

Code: [Select]

#define _Digole_Serial_I2C_
#include <DigoleSerial.h>
#include <Wire.h>
DigoleSerialDisp mydisp(&Wire, '\x27');
int i, j, k, l = 0, m, n, o, p, q, r, x, y, split = 0, nmea = 0,  touchinput = 23, splitsim = 30,  miliseconds = 0, seconds0 = 0, seconds1 = 0, minutes = 0;
char sentence[80];
float  latitude = 0, longitude = 0, speedkmh = 0;
long  laptime = 0, starttime = 0, lastlap = 0, laptimes[100];

void deletescreen()
{
  mydisp.clearScreen();
  mydisp.setRotation(1);
  mydisp.setColor(20);
  mydisp.drawFrame(0, 0, 318, 30);
  mydisp.setColor(1);
  mydisp.setFont(18);
  mydisp.setPrintPos(1, 0, _TEXT_);
  mydisp.setTextPosOffset(0, 9);
  mydisp.print("TEST");
  mydisp.setFont(0);
}

void setup() {
  pinMode(splitsim, INPUT); //simulate split
  digitalWrite(splitsim, HIGH);
  Serial.begin(115200);
  Serial1.begin(115200);
  mydisp.begin();
  deletescreen();
  starttime = millis();

}

void loop() {

  if (Serial1.available())
  {
    char ch = Serial1.read();
    Serial.write(ch);
    if (ch != '\n')
    {
      sentence[i] = ch;
      i++;
    }
    else
    {
      i = 0;
      nmea++;

      //////////////////////////////////////////////////////////////////////////////////////////////////////////////

      latitude = (byte(sentence[20]) - 48) * 10 + (byte(sentence[21]) - 48) + (float(
                   (long(sentence[22]) - 48) * 100000
                   + (long(sentence[23]) - 48) * 10000
                   + (long(sentence[25])  - 48) * 1000
                   + (long(sentence[26]) - 48) * 100
                   + (long(sentence[27]) - 48 ) * 10
                   + (long(sentence[28]) - 48 )) / float(600000));

      longitude = (byte(sentence[33]) - 48) * 10 + (byte(sentence[34]) - 48) + (float(
                    (long(sentence[35]) - 48) * 100000
                    + (long(sentence[36]) - 48) * 10000
                    + (long(sentence[38]) - 48 ) * 1000
                    + (long(sentence[39]) - 48 ) * 100
                    + (long(sentence[40]) - 48 ) * 10
                    + (long(sentence[41]) - 48 )) / float(600000));

      speedkmh = float((long(sentence[45]) - 48 ) * 1000
                       + (long(sentence[46]) - 48 ) * 100
                       + (long(sentence[47]) - 48 ) * 10
                       + (long(sentence[49]) - 48 )) * float(0.1852);
    }
      //////////////////////////////////////////////////////////////////////////////////////////////////////////////

      laptime = millis() - starttime;
      miliseconds = laptime % 1000;
      seconds0 = (laptime % 10000) / 1000;
      seconds1 = (laptime / 10000) % 6;
      minutes = laptime / 60000;

      if (seconds0 == 1)
      {
        split = 0;
      }

      if (digitalRead(splitsim) == 0 && split == 0) {
        split = 1;
        laptimes[l] = laptime;
        l++;
        starttime = millis();
      }
   

    //////////////////////////////////////////////////////////////////////////////////////////////////////////////

    if (nmea % 20 == 0)
    {

      mydisp.setPrintPos(0, 6, _TEXT_);
        mydisp.print("Lap:");
        mydisp.setColor(31);
        if (minutes < 10)
        {
          mydisp.setPrintPos(5, 6, _TEXT_);
          mydisp.print("0");
          mydisp.setPrintPos(6, 6, _TEXT_);
          mydisp.print(minutes);
        }
        else
        {
          mydisp.setPrintPos(5, 6, _TEXT_);
          mydisp.print(minutes);
        }
        mydisp.setPrintPos(7, 6, _TEXT_);
        mydisp.print(":");
        mydisp.setPrintPos(8, 6, _TEXT_);
        mydisp.print(seconds1);
        mydisp.setPrintPos(9, 6, _TEXT_);
        mydisp.print(seconds0);
        mydisp.setPrintPos(10, 6, _TEXT_);
        mydisp.print(":");
        if (miliseconds > 99)
        {
          mydisp.setPrintPos(11, 6, _TEXT_);
          mydisp.print(miliseconds);
        }
        else if (miliseconds > 9 && miliseconds <= 99)
        {
          mydisp.setPrintPos(11, 6, _TEXT_);
          mydisp.print("0");
          mydisp.setPrintPos(12, 6, _TEXT_);
          mydisp.print(miliseconds);
        }
        mydisp.setColor(1);
       
      mydisp.setPrintPos(0, 3, _TEXT_);
      mydisp.print("lat:");
      mydisp.setColor(240);
      mydisp.setPrintPos(5, 3, _TEXT_);
      mydisp.print(latitude, 6);
      mydisp.setColor(1);

      mydisp.setPrintPos(0, 4, _TEXT_);
      mydisp.print("lon:");
      mydisp.setColor(240);
      mydisp.setPrintPos(5, 4, _TEXT_);
      mydisp.print(longitude, 6);
      mydisp.setColor(1);

      mydisp.setPrintPos(16, 3, _TEXT_);
      mydisp.print("data:");
      mydisp.setPrintPos(22, 3, _TEXT_);
      mydisp.print(nmea);

      mydisp.setPrintPos(31, 3, _TEXT_);
      mydisp.print("Kmh:    ");
      mydisp.setColor(60);
      mydisp.setPrintPos(36, 3, _TEXT_);
      mydisp.print(round(speedkmh));
      mydisp.setColor(1);

      mydisp.setPrintPos(19, 6, _TEXT_);
      mydisp.print("Laps:");
      mydisp.setColor(160);
      mydisp.setPrintPos(25, 6, _TEXT_);
      mydisp.print(l);

      mydisp.setColor(1);
    }
   
  }
}


[fuzion]

Doesn't work, serial monitor: mon2.jpg

Code: [Select]

#define _Digole_Serial_I2C_
#include <DigoleSerial.h>
#include <Wire.h>
DigoleSerialDisp mydisp(&Wire, '\x27');
int i, j, k, l = 0, m, n, o, p, q, r, x, y, split = 0, nmea = 0,  touchinput = 23, splitsim = 30,  miliseconds = 0, seconds0 = 0, seconds1 = 0, minutes = 0;
char sentence[80];
float  latitude = 0, longitude = 0, speedkmh = 0;
long  laptime = 0, starttime = 0, lastlap = 0, laptimes[100];

void deletescreen()
{
  mydisp.clearScreen();
  mydisp.setRotation(1);
  mydisp.setColor(20);
  mydisp.drawFrame(0, 0, 318, 30);
  mydisp.setColor(1);
  mydisp.setFont(18);
  mydisp.setPrintPos(1, 0, _TEXT_);
  mydisp.setTextPosOffset(0, 9);
  mydisp.print("TEST");
  mydisp.setFont(0);
}

void setup() {
  pinMode(splitsim, INPUT); //simulate split
  digitalWrite(splitsim, HIGH);
  Serial.begin(115200);
  Serial1.begin(115200);
  mydisp.begin();
  deletescreen();
  starttime = millis();

}

void loop() {

  if (Serial1.available())
  {
    char ch = Serial1.read();
    Serial.write(ch);
    if (ch != '\n')
    {
      sentence[i] = ch;
      i++;
    }
    else
    {
      i = 0;
      nmea++;

      //////////////////////////////////////////////////////////////////////////////////////////////////////////////

      latitude = (byte(sentence[20]) - 48) * 10 + (byte(sentence[21]) - 48) + (float(
                   (long(sentence[22]) - 48) * 100000
                   + (long(sentence[23]) - 48) * 10000
                   + (long(sentence[25])  - 48) * 1000
                   + (long(sentence[26]) - 48) * 100
                   + (long(sentence[27]) - 48 ) * 10
                   + (long(sentence[28]) - 48 )) / float(600000));

      longitude = (byte(sentence[33]) - 48) * 10 + (byte(sentence[34]) - 48) + (float(
                    (long(sentence[35]) - 48) * 100000
                    + (long(sentence[36]) - 48) * 10000
                    + (long(sentence[38]) - 48 ) * 1000
                    + (long(sentence[39]) - 48 ) * 100
                    + (long(sentence[40]) - 48 ) * 10
                    + (long(sentence[41]) - 48 )) / float(600000));

      speedkmh = float((long(sentence[45]) - 48 ) * 1000
                       + (long(sentence[46]) - 48 ) * 100
                       + (long(sentence[47]) - 48 ) * 10
                       + (long(sentence[49]) - 48 )) * float(0.1852);

      //////////////////////////////////////////////////////////////////////////////////////////////////////////////

      laptime = millis() - starttime;
      miliseconds = laptime % 1000;
      seconds0 = (laptime % 10000) / 1000;
      seconds1 = (laptime / 10000) % 6;
      minutes = laptime / 60000;

      if (seconds0 == 1)
      {
        split = 0;
      }

      if (digitalRead(splitsim) == 0 && split == 0) {
        split = 1;
        laptimes[l] = laptime;
        l++;
        starttime = millis();
      }
    

    //////////////////////////////////////////////////////////////////////////////////////////////////////////////

    if (nmea % 20 == 0)
    {

      mydisp.setPrintPos(0, 6, _TEXT_);
        mydisp.print("Lap:");
        mydisp.setColor(31);
        if (minutes < 10)
        {
          mydisp.setPrintPos(5, 6, _TEXT_);
          mydisp.print("0");
          mydisp.setPrintPos(6, 6, _TEXT_);
          mydisp.print(minutes);
        }
        else
        {
          mydisp.setPrintPos(5, 6, _TEXT_);
          mydisp.print(minutes);
        }
        mydisp.setPrintPos(7, 6, _TEXT_);
        mydisp.print(":");
        mydisp.setPrintPos(8, 6, _TEXT_);
        mydisp.print(seconds1);
        mydisp.setPrintPos(9, 6, _TEXT_);
        mydisp.print(seconds0);
        mydisp.setPrintPos(10, 6, _TEXT_);
        mydisp.print(":");
        if (miliseconds > 99)
        {
          mydisp.setPrintPos(11, 6, _TEXT_);
          mydisp.print(miliseconds);
        }
        else if (miliseconds > 9 && miliseconds <= 99)
        {
          mydisp.setPrintPos(11, 6, _TEXT_);
          mydisp.print("0");
          mydisp.setPrintPos(12, 6, _TEXT_);
          mydisp.print(miliseconds);
        }
        mydisp.setColor(1);
        
      mydisp.setPrintPos(0, 3, _TEXT_);
      mydisp.print("lat:");
      mydisp.setColor(240);
      mydisp.setPrintPos(5, 3, _TEXT_);
      mydisp.print(latitude, 6);
      mydisp.setColor(1);

      mydisp.setPrintPos(0, 4, _TEXT_);
      mydisp.print("lon:");
      mydisp.setColor(240);
      mydisp.setPrintPos(5, 4, _TEXT_);
      mydisp.print(longitude, 6);
      mydisp.setColor(1);

      mydisp.setPrintPos(16, 3, _TEXT_);
      mydisp.print("data:");
      mydisp.setPrintPos(22, 3, _TEXT_);
      mydisp.print(nmea);

      mydisp.setPrintPos(31, 3, _TEXT_);
      mydisp.print("Kmh:    ");
      mydisp.setColor(60);
      mydisp.setPrintPos(36, 3, _TEXT_);
      mydisp.print(round(speedkmh));
      mydisp.setColor(1);

      mydisp.setPrintPos(19, 6, _TEXT_);
      mydisp.print("Laps:");
      mydisp.setColor(160);
      mydisp.setPrintPos(25, 6, _TEXT_);
      mydisp.print(l);

      mydisp.setColor(1);
    }
    }
  }
}


[-dev]

Lots of tips in this thread and the project it links to, by Shift314.

Basically, use NeoGPS. It's smaller, faster and more accurate than all other libraries. It can be configured to parse only the speed and date/time, saving even more Ram, program space and execution time.

If you want to try it, it's available from the IDE Library Manager, in the Sketch->Include Library -> Manage Libraries.

[fuzion]

seems interesting, i ll definitely try it thanks

[Robin2]

ok did it now. the only thing i change is the bracket of else statement

You have almost the entire program in the loop() function. It is too difficult to figure out what's what. If you break the program into separate short single purpose functions the problem will probably be very obvious.

...R

[fuzion]

i understand that functioning will be necessary, but this program is still too small for it to be not understandable.

i read neogps a bit, it seems to be faster than other libraries, but i am not sure it will be faster than a simple char read and use of the 10-20 specific chars someone wants. anyway i ll give it a shot.

what i also found out is that i only need the gll sentence instead of the rmc, which will save me some processing time.

[-dev]

i only need the gll sentence instead of the rmc

You don't need speed?

i am not sure it will be faster than a simple char read and use of the 10-20 specific chars someone wants.

That's a good question.  Surprisingly, I think it will be faster for your case, because of buffering, parsing and math.  NeoGPS performs incremental parsing, so the parsing and math CPU time is spread out over the 50ms interval.  The fix information is completely assembled by the time the '\n' arrives.

NeoGPS also avoids floating-point math.  The example NMEAloc.ino shows how to print the lat/lon with integers... it just sneaks a '.' into the output stream at the right digit, and it looks like a floating-point number (and with more significant digits).  This is also much faster than printing a floating-point number.  Since you're rounding the speed to an integer, it is possible that the entire floating-point library could be eliminated.

One of the NeoGPS programs is NMEA_isr.ino.  It handles the GPS characters during the RX char interrupt.  This avoids saving it in the Serial1 buffer, copying it out into your buffer (with read()), then parsing your buffer all at once.  This also makes it immune to losing GPS characters.  If your screen update takes a little too long, the GPS characters will still get handled in the background.  When loop executes again, the fix will be waiting to be read().  In fact, the fix will be available 174us sooner (two character times), and the overall CPU utilization will be lower.

With these two techniques, NeoGPS incremental parsing and NeoHWSerial background processing, I'll bet your UNO could have worked.  You would have had to put the GPS device on Serial (GPS TX to UNO RX pin 0), but it might have worked with NeoICSerial (GPS TX pin connected to UNO pin 8).

Other notes:

*   Comparing the two running sketches would be an interesting exercise, but would have to be performed with two UNOs and an oscilloscope or logic analyzer.  A few extra digital output pins could be used to measure various sections of the sketch.

*   powergravity found that the reporting frequency is stated to be 20Hz, but the sentences arrive at irregular intervals between 10ms and 90ms.  He did not analyze the data to determine if the fixes are "spaced" regularly (locations vs. speed).  First thread here.

*   Since this is a vehicle, its mass really keeps it going in a straight line.  This means that a lower update rate is probably accurate, because the linear interpolation between 5Hz updates has to be close to any possible vehicle curve (i.e., changes in velocity).  You might get better system behavior at lower rates, without losing very much vehicle position accuracy.  That is, as long as you're NOT recording an impact, where accelerations are orders of magnitude higher than the engine or steering can normally produce.  o_O

*   Make sure you have disabled sentences that you don't use, with the Configure NMEA binary command.  This will reduce the number of interrupts that the Arduino has to handle.

*   I have attached a NeoGPS version of your sketch.  I changed all the double-quoted strings to use the F macro (saves RAM), but I wasn't sure if your Digole library supports that.  You'll get a compile error about "FlashStringHelper" if it doesn't.

Cheers,
/dev

[Robin2]

but this program is still too small for it to be not understandable..

According to whom?

...R
(PS  not me)

[fuzion]

according to me, i wrote the sentence

dev thanks. i will look into the program and test simple things with neogps to see the difference and if it helps. if it indeed is built around saving processing time during serial read etc. it will save the day.

I should note that at 10Hz - 38400 i have no such problems, therefore one could assume that it has something to do with processing time, but on the other hand this seems far fetched.

- no i dont need speed. i just need position, to check passing from track finish lines. Actually i only need the position and nothing else. yes i have disabled all sentences but the one i need. But, sparkfun was not kind, they update with 1hz only the GLL sentence. stuck on rmc for now, i will email them for a firmware the support is very good.

- i will need some basic floating point math (or at least long) to calculate degrees difference between live and finish lines lat-long. chars and tricks with . i think will be of no use to me here.

- the theory behind 5hz could be correct with the gps correlation. at some point if i don't solve the 20hz problems, i will go back to 10, everything was much easier.

- the irregular intervals of the sentences could be a reality, i read about it on other forums as well. i see on my laptimes that there is at least 1-15ms difference, as i do things on the \n character and millis() i should always get 50ms interval times, this is not happening.

- do you think that the I2C protocol for the screen messes with the serial somehow? it also supports uart and spi.

i will post back when i do some tests with neogps. perhaps in a day or two.

[Robin2]

according to me, i wrote the sentence

I am old enough to know that my opinions about me and my work don't really count for anything. (Including this one)

...R