GPS data analysis

Using Arduino Programming Questions
1

Hi,

I'm trying to analyse how often (and how much) data a GPS module is generating. I started with this very simple loop.....

    for (int i=0 ;i<36000; i++)
    {
      bytesRead = 0;
      while (Serial1.available())
      {
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
      Serial.print("Read <");Serial.print(bytesRead);Serial.print("> bytes from GPS port at <");Serial.print(millis());Serial.println("> milli seconds since startup.");
      bytesRead = 0;
    }

this gives no problems and gives me a result like this.....

Read <0> bytes from GPS port at <1433> milli seconds since startup.
Read <0> bytes from GPS port at <1438> milli seconds since startup.
Read <0> bytes from GPS port at <1444> milli seconds since startup.
Read <35> bytes from GPS port at <1451> milli seconds since startup.
Read <34> bytes from GPS port at <1457> milli seconds since startup.
Read <34> bytes from GPS port at <1463> milli seconds since startup.
Read <33> bytes from GPS port at <1468> milli seconds since startup.
Read <34> bytes from GPS port at <1474> milli seconds since startup.
Read <34> bytes from GPS port at <1480> milli seconds since startup.
Read <33> bytes from GPS port at <1486> milli seconds since startup.
Read <34> bytes from GPS port at <1492> milli seconds since startup.
Read <33> bytes from GPS port at <1498> milli seconds since startup.
Read <34> bytes from GPS port at <1504> milli seconds since startup.
Read <34> bytes from GPS port at <1510> milli seconds since startup.
Read <33> bytes from GPS port at <1516> milli seconds since startup.
Read <11> bytes from GPS port at <1522> milli seconds since startup.
Read <0> bytes from GPS port at <1527> milli seconds since startup.
Read <0> bytes from GPS port at <1533> milli seconds since startup.
Read <0> bytes from GPS port at <1540> milli seconds since startup.

Which is exactly what was expected - it tells me at what moment the GPS sent a burst of data.

I then decided to modify the routine to totalise the amount of data received in each burst. this is what I wrote....

    for (int i=0 ;i<36000; i++)
    {
      while (Serial1.available())
      {
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
      
      // totalise the amount of data received in each 'burst'
      if ( bytesRead != 0 )
      {
        totBytesRead = totBytesRead + bytesRead;
        burst++;
      }
      
      Serial.print("bytesRead=<");Serial.print(bytesRead);Serial.print(">, totBytesRead=<");Serial.print(totBytesRead);Serial.print(">, burst=<");Serial.print(burst);Serial.print("> prevBytesRead=<");Serial.print(prevBytesRead);Serial.println(">");
      
      if( (bytesRead == 0) && (prevBytesRead!=0) )
      {
        Serial.print("Read <");Serial.print(totBytesRead);Serial.print(">bytes, in <");Serial.print(burst);Serial.print(">bursts, from port at <");Serial.print(millis());Serial.println("> milli secs since start.");
        totBytesRead=0;
        burst=0;
      }
      
      prevBytesRead = bytesRead;
      bytesRead = 0;
    }

As before, this produced exactly what I expected.....

bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<3>, totBytesRead=<3>, burst=<1> prevBytesRead=<0>
bytesRead=<32>, totBytesRead=<35>, burst=<2> prevBytesRead=<3>
bytesRead=<30>, totBytesRead=<65>, burst=<3> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<97>, burst=<4> prevBytesRead=<30>
bytesRead=<31>, totBytesRead=<128>, burst=<5> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<160>, burst=<6> prevBytesRead=<31>
bytesRead=<31>, totBytesRead=<191>, burst=<7> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<223>, burst=<8> prevBytesRead=<31>
bytesRead=<32>, totBytesRead=<255>, burst=<9> prevBytesRead=<32>
bytesRead=<31>, totBytesRead=<286>, burst=<10> prevBytesRead=<32>
bytesRead=<33>, totBytesRead=<319>, burst=<11> prevBytesRead=<31>
bytesRead=<32>, totBytesRead=<351>, burst=<12> prevBytesRead=<33>
bytesRead=<32>, totBytesRead=<383>, burst=<13> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<415>, burst=<14> prevBytesRead=<32>
bytesRead=<6>, totBytesRead=<421>, burst=<15> prevBytesRead=<32>
bytesRead=<0>, totBytesRead=<421>, burst=<15> prevBytesRead=<6>
Read <421>bytes, in <15>bursts, from port at <3142> milli secs since start.

Ok, now I'm getting to the hub of the matter. The only part of the output I'm actually interested in is the line...

Read <421>bytes, in <15>bursts, from port at <3142> milli secs since start

So, all I need to do is comment out this line...

      Serial.print("bytesRead=<");Serial.print(bytesRead);Serial.print(">, totBytesRead=<");Serial.print(totBytesRead);Serial.print(">, burst=<");Serial.print(burst);Serial.print("> prevBytesRead=<");Serial.print(prevBytesRead);Serial.println(">");

However, as soon as I remove this line (which does nothing but display to the screen) the results actually get changed.
This is what I get.....

Read <1>bytes, in <1>bursts, from port at <1868> milli secs since start.
Read <18>bytes, in <2>bursts, from port at <1871> milli secs since start.
Read <24>bytes, in <1>bursts, from port at <1878> milli secs since start.
Read <37>bytes, in <1>bursts, from port at <1884> milli secs since start.
Read <35>bytes, in <1>bursts, from port at <1890> milli secs since start.
Read <37>bytes, in <1>bursts, from port at <1896> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1903> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1909> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1915> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1922> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1929> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1935> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1941> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1947> milli secs since start.
Read <15>bytes, in <1>bursts, from port at <1953> milli secs since start.
Read <1>bytes, in <1>bursts, from port at <2068> milli secs since start.
Read <18>bytes, in <1>bursts, from port at <2071> milli secs since start.

Compared to what I expected....

Read <470>bytes, in <16>bursts, from port at <211> milli secs since start.
Read <470>bytes, in <15>bursts, from port at <408> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <610> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <812> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1008> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1210> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1413> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1609> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1812> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <2009> milli secs since start.

On the plus side, I've actually got the results I was after by hand-editing the screen output, but on the downside there's obviously something going wrong that I've missed. How can the inclusion of a series of print() statements effect the actual results? and how can I cure it.

Thanks

2

Look over what a GPS typically sends to the host:
http://www.maartenlamers.com/nmea/

Your character streams are non-blocked and are uniquely formatted. Generally, we in the Arduino community use a GPS library to parse what is needed. Some write their own routines. Check out the standard libraries: http://playground.arduino.cc//Main/LibraryList
or use Google to get a larger list.

Ray

3

I'm gonna go tru you lines of code.

But One thing is for sure. Programming every thing on a single line look good in you code. But it make it harder to debug when there is a problem. May I suggest to do a multi-line until everything work?

4

Thanks mrburnette

I know what a GPS typically sends, and I do have a functioning GPS library. I've been investigating how often (and how many) characters are been generated by the GPS - at this moment I'm not actually interested in what the data is.

The problem is this - the code shown above works fine when I include a set of print() statements, but as soon as I comment out that line, the actual results change!

With the print() line enabled I see that I get a block of about 470 characters delivered in 16 'bursts' of data - but as soon as I remove the print() line it changes to show me getting about 30-40 bytes delivered in a single block. Why? the print() statements shouldn't change anything

Cheers

5

Ok, let's see

Serial.print("bytesRead=<");
Serial.print(bytesRead);
Serial.print(">, totBytesRead=<");
Serial.print(totBytesRead);
Serial.print(">, burst=<");
Serial.print(burst);
Serial.print("> prevBytesRead=<");
Serial.print(prevBytesRead);
Serial.println(">");
6

I'm gonna go tru you lines of code.

Thanks

I only multi-line the print() statements - didn't see much harm in it myself.

cheers

7

Yeah that's what i'm seeing to. :wink:

But, you can't catch fly with vinegar, so if some one see that he is gonna have to scroll left and right to understand your stuff, he might give up before even noticing that it's not the important part.

You know that it's not important, but we don't, until we read it. And at first look, it could scare the less courageous one that could still find the answer right away if you sell your stuff right.

8

The lines you are printing are quite long. The serial output buffer is (IIRC) 32 bytes, so the output will block until there's less than 32 bytes in the buffer. Changing the amount of data printed changes the timing of when, how often and for how long it blocks.
Simplest thing is to change the serial output speed (and the serial monitor) to 57600 or even higher.

Pete

9

The serial monitor is already set to 115200;

I've got to admit that I've never had a problem because I'm printing out too much!

thanls

10

Change the output to print only the two numbers separated by a space. See if that changes and/or helps things.

Pete

11 
      if( (bytesRead == 0) && (prevBytesRead!=0) )
      {
        Serial.print("Read <");Serial.print(totBytesRead);Serial.print(">bytes, in <");Serial.print(burst);Serial.print(">bursts, from port at <");Serial.print(millis());Serial.println("> milli secs since start.");
        totBytesRead=0;
        burst=0;
      }
      
      prevBytesRead = bytesRead;
      bytesRead = 0;
    }

I may be way off base here (and not at all properly explaining what I'm seeing), but....

Two things —

Looks like you're resetting totBytesRead, burst, & bytesRead to 0 after printing the values, no?

Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?

12

Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?

This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long. Basically the delay was allowing more time for the incoming data to be queued up at the input port for processing.

well, I know who to go to first next time I have a problem.

Thanks for everyone's contributions.

13

OhMyCod:

Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?

This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long. Basically the delay was allowing more time for the incoming data to be queued up at the input port for processing.

well, I know who to go to first next time I have a problem.

Thanks for everyone's contributions.

Sweet!

Yep....the wife always knows. :roll_eyes:

14

OhMyCod:
This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long.

Adding a delay "solves" it but the wrong way. Read this:

This is wrong, for example:

while (Serial1.available())

{
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
      Serial.print("Read <");Serial.print(bytesRead);Serial.print("> bytes from GPS port at <");Serial.print(millis());Serial.println("> milli seconds since startup.");

You are printing even if there is nothing available. Clearly the print should be inside the "while available" not outside.

15

Even that isn't quite right because it would print a line for every received byte.

Read the link about doing serial reads, loops with delays usually are the wrong way of doing it.

16

You are printing even if there is nothing available.

This is correct - but this is what I want to do - I'm trying to analise when data is arriving (and when it's not arriving!)

Adding a delay "solves" it but the wrong way.

Agreed. more importantly it explains what I was seeing.

Read this:

I did. good link.

The work I've done has showed me that the GPS transmits a 'burst' about half a kilobyte of data (consisting of 6 different NMEA records) followed by a pause of several milliseconds before transmistting another burst.

Still haven't worked out why it transmits the records in what seems to be a continuously changing random order, but that's another question.

Cheers