Pages: [1] 2   Go Down
Author Topic: GPS data analysis  (Read 516 times)
0 Members and 1 Guest are viewing this topic.
Offline Offline
Full Member
***
Karma: 1
Posts: 151
What could possibly go wrong?
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Hi,

I'm trying to analyse how often (and how much) data a GPS module is generating. I started with this very simple loop.....
Code:
    for (int i=0 ;i<36000; i++)
    {
      bytesRead = 0;
      while (Serial1.available())
      {
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
      Serial.print("Read <");Serial.print(bytesRead);Serial.print("> bytes from GPS port at <");Serial.print(millis());Serial.println("> milli seconds since startup.");
      bytesRead = 0;
    }

this gives no problems and gives me a result like this.....
Quote
Read <0> bytes from GPS port at <1433> milli seconds since startup.
Read <0> bytes from GPS port at <1438> milli seconds since startup.
Read <0> bytes from GPS port at <1444> milli seconds since startup.
Read <35> bytes from GPS port at <1451> milli seconds since startup.
Read <34> bytes from GPS port at <1457> milli seconds since startup.
Read <34> bytes from GPS port at <1463> milli seconds since startup.
Read <33> bytes from GPS port at <1468> milli seconds since startup.
Read <34> bytes from GPS port at <1474> milli seconds since startup.
Read <34> bytes from GPS port at <1480> milli seconds since startup.
Read <33> bytes from GPS port at <1486> milli seconds since startup.
Read <34> bytes from GPS port at <1492> milli seconds since startup.
Read <33> bytes from GPS port at <1498> milli seconds since startup.
Read <34> bytes from GPS port at <1504> milli seconds since startup.
Read <34> bytes from GPS port at <1510> milli seconds since startup.
Read <33> bytes from GPS port at <1516> milli seconds since startup.
Read <11> bytes from GPS port at <1522> milli seconds since startup.
Read <0> bytes from GPS port at <1527> milli seconds since startup.
Read <0> bytes from GPS port at <1533> milli seconds since startup.
Read <0> bytes from GPS port at <1540> milli seconds since startup.

Which is exactly what was expected - it tells me at what moment the GPS sent a burst of data.

I then decided to modify the routine to totalise the amount of data received in each burst. this is what I wrote....
Code:
    for (int i=0 ;i<36000; i++)
    {
      while (Serial1.available())
      {
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
     
      // totalise the amount of data received in each 'burst'
      if ( bytesRead != 0 )
      {
        totBytesRead = totBytesRead + bytesRead;
        burst++;
      }
     
      Serial.print("bytesRead=<");Serial.print(bytesRead);Serial.print(">, totBytesRead=<");Serial.print(totBytesRead);Serial.print(">, burst=<");Serial.print(burst);Serial.print("> prevBytesRead=<");Serial.print(prevBytesRead);Serial.println(">");
     
      if( (bytesRead == 0) && (prevBytesRead!=0) )
      {
        Serial.print("Read <");Serial.print(totBytesRead);Serial.print(">bytes, in <");Serial.print(burst);Serial.print(">bursts, from port at <");Serial.print(millis());Serial.println("> milli secs since start.");
        totBytesRead=0;
        burst=0;
      }
     
      prevBytesRead = bytesRead;
      bytesRead = 0;
    }

As before, this produced exactly what I expected.....
Quote
bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<0>, totBytesRead=<0>, burst=<0> prevBytesRead=<0>
bytesRead=<3>, totBytesRead=<3>, burst=<1> prevBytesRead=<0>
bytesRead=<32>, totBytesRead=<35>, burst=<2> prevBytesRead=<3>
bytesRead=<30>, totBytesRead=<65>, burst=<3> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<97>, burst=<4> prevBytesRead=<30>
bytesRead=<31>, totBytesRead=<128>, burst=<5> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<160>, burst=<6> prevBytesRead=<31>
bytesRead=<31>, totBytesRead=<191>, burst=<7> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<223>, burst=<8> prevBytesRead=<31>
bytesRead=<32>, totBytesRead=<255>, burst=<9> prevBytesRead=<32>
bytesRead=<31>, totBytesRead=<286>, burst=<10> prevBytesRead=<32>
bytesRead=<33>, totBytesRead=<319>, burst=<11> prevBytesRead=<31>
bytesRead=<32>, totBytesRead=<351>, burst=<12> prevBytesRead=<33>
bytesRead=<32>, totBytesRead=<383>, burst=<13> prevBytesRead=<32>
bytesRead=<32>, totBytesRead=<415>, burst=<14> prevBytesRead=<32>
bytesRead=<6>, totBytesRead=<421>, burst=<15> prevBytesRead=<32>
bytesRead=<0>, totBytesRead=<421>, burst=<15> prevBytesRead=<6>
Read <421>bytes, in <15>bursts, from port at <3142> milli secs since start.

Ok, now I'm getting to the hub of the matter. The only part of the output I'm actually interested in is the line...
Quote
Read <421>bytes, in <15>bursts, from port at <3142> milli secs since start

So, all I need to do is comment out this line...
Code:
      Serial.print("bytesRead=<");Serial.print(bytesRead);Serial.print(">, totBytesRead=<");Serial.print(totBytesRead);Serial.print(">, burst=<");Serial.print(burst);Serial.print("> prevBytesRead=<");Serial.print(prevBytesRead);Serial.println(">");

However, as soon as I remove this line (which does nothing but display to the screen) the results actually get changed.
This is what I get.....
Quote
Read <1>bytes, in <1>bursts, from port at <1868> milli secs since start.
Read <18>bytes, in <2>bursts, from port at <1871> milli secs since start.
Read <24>bytes, in <1>bursts, from port at <1878> milli secs since start.
Read <37>bytes, in <1>bursts, from port at <1884> milli secs since start.
Read <35>bytes, in <1>bursts, from port at <1890> milli secs since start.
Read <37>bytes, in <1>bursts, from port at <1896> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1903> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1909> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1915> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1922> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1929> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1935> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1941> milli secs since start.
Read <36>bytes, in <1>bursts, from port at <1947> milli secs since start.
Read <15>bytes, in <1>bursts, from port at <1953> milli secs since start.
Read <1>bytes, in <1>bursts, from port at <2068> milli secs since start.
Read <18>bytes, in <1>bursts, from port at <2071> milli secs since start.

Compared to what I expected....
Quote
Read <470>bytes, in <16>bursts, from port at <211> milli secs since start.
Read <470>bytes, in <15>bursts, from port at <408> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <610> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <812> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1008> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1210> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1413> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1609> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <1812> milli secs since start.
Read <470>bytes, in <16>bursts, from port at <2009> milli secs since start.

On the plus side, I've actually got the results I was after by hand-editing the screen output, but on the downside there's obviously something going wrong that I've missed. How can the inclusion of a series of print() statements effect the actual results? and how can I cure it.

Thanks
Logged

Atlanta, USA
Offline Offline
Edison Member
*
Karma: 33
Posts: 1424
AKA: Ray Burne
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Look over what a GPS typically sends to the host:
http://www.maartenlamers.com/nmea/

Your character streams are non-blocked and are uniquely formatted.  Generally, we in the Arduino community use a GPS library to parse what is needed.  Some write their own routines.  Check out the standard libraries: http://playground.arduino.cc//Main/LibraryList
or use Google to get a larger list.


Ray
Logged

Québec
Offline Offline
Sr. Member
****
Karma: 7
Posts: 388
When it's not human error, it's Windows going schizophrenic.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

I'm gonna go tru you lines of code.

But One thing is for sure. Programming every thing on a single line look good in you code. But it make it harder to debug when there is a problem. May I suggest to do a multi-line until everything work?
Logged

Offline Offline
Full Member
***
Karma: 1
Posts: 151
What could possibly go wrong?
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Thanks mrburnette

I know what a GPS typically sends, and I do have a functioning GPS library. I've been investigating how often (and how many) characters are been generated by the GPS - at this moment I'm not actually interested in what the data is.

The problem is this - the code shown above works fine when I include a set of print() statements, but as soon as I comment out that line, the actual results change!

With the print() line enabled I see that I get a block of about 470 characters delivered in 16 'bursts' of data - but as soon as I remove the print() line it changes to show me getting about 30-40 bytes delivered in a single block. Why? the print() statements shouldn't change anything

Cheers
Logged

Québec
Offline Offline
Sr. Member
****
Karma: 7
Posts: 388
When it's not human error, it's Windows going schizophrenic.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Ok, let's see
Code:
Serial.print("bytesRead=<");
Serial.print(bytesRead);
Serial.print(">, totBytesRead=<");
Serial.print(totBytesRead);
Serial.print(">, burst=<");
Serial.print(burst);
Serial.print("> prevBytesRead=<");
Serial.print(prevBytesRead);
Serial.println(">");
Logged

Offline Offline
Full Member
***
Karma: 1
Posts: 151
What could possibly go wrong?
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
I'm gonna go tru you lines of code.
Thanks

I only multi-line the print() statements - didn't see much harm in it myself.

cheers
Logged

Québec
Offline Offline
Sr. Member
****
Karma: 7
Posts: 388
When it's not human error, it's Windows going schizophrenic.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Yeah that's what i'm seeing to. smiley-wink

But, you can't catch fly with vinegar, so if some one see that he is gonna have to scroll left and right to understand your stuff, he might give up before even noticing that it's not the important part.

You know that it's not important, but we don't, until we read it. And at first look, it could scare the less courageous one that could still find the answer right away if you sell your stuff right.
Logged

Offline Offline
Edison Member
*
Karma: 31
Posts: 1417
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The lines you are printing are quite long. The serial output buffer is (IIRC) 32 bytes, so the output will block until there's less than 32 bytes in the buffer. Changing the amount of data printed changes the timing of when, how often and for how long it blocks.
Simplest thing is to change the serial output speed (and the serial monitor) to 57600 or even higher.

Pete
Logged

Where are the Nick Gammons of yesteryear?

Offline Offline
Full Member
***
Karma: 1
Posts: 151
What could possibly go wrong?
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

The serial monitor is already set to 115200;

I've got to admit that I've never had a problem because I'm printing out too much!

thanls
Logged

Offline Offline
Edison Member
*
Karma: 31
Posts: 1417
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Change the output to print only the two numbers separated by a space. See if that changes and/or helps things.

Pete
Logged

Where are the Nick Gammons of yesteryear?

Chicago
Offline Offline
Sr. Member
****
Karma: 5
Posts: 456
With every answer comes more questions.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset




Code:
      if( (bytesRead == 0) && (prevBytesRead!=0) )
      {
        Serial.print("Read <");Serial.print(totBytesRead);Serial.print(">bytes, in <");Serial.print(burst);Serial.print(">bursts, from port at <");Serial.print(millis());Serial.println("> milli secs since start.");
        totBytesRead=0;
        burst=0;
      }
     
      prevBytesRead = bytesRead;
      bytesRead = 0;
    }


I may be way off base here (and not at all properly explaining what I'm seeing), but....

Two things —

Looks like you're resetting totBytesRead, burst, & bytesRead to 0 after printing the values, no?

Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?
Logged

Offline Offline
Full Member
***
Karma: 1
Posts: 151
What could possibly go wrong?
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?

This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long. Basically the delay was allowing more time for the incoming data to be queued up at the input port for processing.

well, I know who to go to first next time I have a problem.

Thanks for everyone's contributions.
Logged

Chicago
Offline Offline
Sr. Member
****
Karma: 5
Posts: 456
With every answer comes more questions.
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Quote
Without the extra print statements (to slow things down) could the loop be running through again before more data is sent/received by GPS?

This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long. Basically the delay was allowing more time for the incoming data to be queued up at the input port for processing.

well, I know who to go to first next time I have a problem.

Thanks for everyone's contributions.

Sweet!

Yep....the wife always knows.   smiley-roll
Logged

Global Moderator
Offline Offline
Brattain Member
*****
Karma: 452
Posts: 18694
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

This is correct. The problem was actually solved in about 10 seconds flat by my wife, if I take out the series of print() statements and insert a delay of 5 milliseconds then I get exactly the same results. I knew a print() would be time consuming, but I had no idea it would take so long.

Adding a delay "solves" it but the wrong way. Read this:

http://www.gammon.com.au/serial

This is wrong, for example:

Quote
Code:
while (Serial1.available())
      {
        // Read (and ignore) a SINGLE byte from the port
        Serial1.read();
        bytesRead++;
      }
      Serial.print("Read <");Serial.print(bytesRead);Serial.print("> bytes from GPS port at <");Serial.print(millis());Serial.println("> milli seconds since startup.");

You are printing even if there is nothing available. Clearly the print should be inside the "while available" not outside.
Logged

Global Moderator
Offline Offline
Brattain Member
*****
Karma: 452
Posts: 18694
View Profile
 Bigger Bigger  Smaller Smaller  Reset Reset

Even that isn't quite right because it would print a line for every received byte.

Read the link about doing serial reads, loops with delays usually are the wrong way of doing it.
Logged

Pages: [1] 2   Go Up
Jump to: