Go Down

Topic: NeoGPS - configurable, ultra-small RAM footprint (Read 16813 times) previous topic - next topic

-dev

> Less than what I wanted to know, actually.

You're a glutton for punishment.  Maybe these new README sections would be of interest: Configuration and Extending NeoGPS.  The latter describes three ways to add device-specific behavior.

Cheers,
/dev

jboyton

#16
Jan 06, 2015, 09:15 pm Last Edit: Jan 06, 2015, 09:16 pm by jboyton
Thanks for the extra information. I still had to look at the code to figure out how to print out all of the fields I was interested in, but it wasn't too big of a deal.

I wrote a simple sketch to print out all of the values from the GGA and RMC and it fell together without much effort.

One thing I noticed is that dateTime.Year is returning 45 instead of 15 or 2015.

I like what you've done.

jboyton

#17
Jan 06, 2015, 09:45 pm Last Edit: Jan 06, 2015, 10:47 pm by jboyton
I wanted to do some sort of comparison between NeoGPS and the code I'm using. It's difficult to do a fair apples to apples comparison since it depends on the details of the test. Also, since I am writing code for a specific device and a specific application I can tailor it to the exact requirements.

That said, I took a simple sketch which acquired GGA and RMC strings once per second at 38400 baud and then printed out all of the values.

NeoGPS: object size = 43
my GPS: object size = 86

NeoGPS: sketch = 9950; SRAM = 515
my GPS: sketch = 6532; SRAM = 457


Then I compared parsing time, using the GGA and RMC strings in your benchmark example

NeoGPS: GGA = 1008 us; RMC = 1060 us
my GPS: GGA = 448 us; RMC = 468 us


I was mostly surprised that my code used less ram since it stores most of the data as characters instead of encoding it as integers or floats. But the software serial has a 64 byte buffer, something I was able to do away with in my code by calling the parse function from the ISR with interrupts re-enabled. It might be possible to do the same thing using NeoGPS but it would require some modification to the library.

Apples to apples? Probably not. But at first glance it doesn't look like NeoGPS will save me resources on this particular project. I'm a little disappointed since your library is so much more sophisticated than mine.



-dev

Interesting work!

> One thing I noticed is that dateTime.Year is returning 45 instead of 15 or 2015.

Yes, that Time library has an epoch of 1970.  The Year is really an offset.  I prefer an epoch of 2000, or even <this year>, as date-to-seconds conversion is much quicker.  Thanks, Mr. February!

> NeoGPS: object size = 43
> my GPS: object size = 86


I would suggest disabling the virtuals.  It's not a big effect, though.

> NeoGPS: sketch = 9950; SRAM = 515
> my GPS: sketch = 6532; SRAM = 457


That's a number alright.  Is that a high-water mark?  Well, I can tell you that NeoGPS does not use any more RAM than its object size... except when you call it, and then it's using the stack.  When it returns from the call, that RAM is "returned".  There are many sneaky ways that RAM can get used, so I'd have to look closer to explain what the numbers mean, and whether they are both apples.

Here's something that caught my eye:

  Serial.print("Memory used: "); Serial.println(2048 - freeMemory());

This is executed after a sentence is decoded, so I'm not sure what it's measuring.  And it turns out that string literals use RAM.  For a pleasant surprise, I would suggest replacing all the string literals with F("string literal"), at least the ones that are printed.  Boy, this is a topic in itself.

> Then I compared parsing time...
>
> NeoGPS: GGA = 1008 us; RMC = 1060 us
> my GPS: GGA = 448 us; RMC = 468 us


I think the times actually makes sense, given that uGPS...

> ... stores most of the data as characters instead of endecoding it
> as integers or floats.


Most libraries do not do math very well.  And just to clarify, NeoGPS does not do any floating-point math while parsing.  You can access the integers as if they were floats, because it does a conversion when you ask for it, long after the parsing is done.  These are the "floating-point accessors" referred to in the README.  NeoGPS is very careful about division, modulo (mostly a divide), and 32-bit math operations because they are much slower on an 8-bit processor.  Fortunately, it does have 8- and 16-bit multiply instructions that execute in one cycle.

I originally considered not decoding the integers: the bytes could stored as BCD.  It's fairly dense (two digits per byte) and fairly easy to decode.  The RAM size, however, was still larger than I wanted.  After looking at the assembly, I realized it didn't take any more time to do a multiply-by-ten than doing a left-shift 4 for BCD.

Back to the timings.  I think they're probably right, given the operations needed to convert characters to ints.  Don't even get me started on the funky DDDmm.mmm format!  Sheesh, 1-3 digits of degrees, no separator, then floating-point minutes?!?

It really gets down to whether you need the information from the device, or just the bytes.  If you are comparing values or calculating a distance, you need the numbers.  This is the NeoGPS et al niche.

If you are just logging or displaying the bytes, perhaps in a different order, you only need... bytes.  A much simpler problem.

> ...by calling the parse function from the ISR...

Hey!  That's exactly the async mode I mentioned in the README.  I usually work in the Cosa environment, and it's very easy to hook `decode` to the ISR.  You have to be careful because the fix data is volatile when it's populated by an ISR.

> ...with interrupts re-enabled.

:o   You sir, are a square peg.  :D

> I'm a little disappointed since your library is so much more sophisticated than mine.

Bahahaha, "sophisticated."  Wait, that's not always a good thing.    :smiley-confuse:

Unless you're parsing those bytes later, it sounds like NeoGPS isn't really a good fit.  If you do parse the bytes later, you're really just buffering the data like most other libraries.  Then you have to add the SRAM and CPU time numbers because you're doing both things: buffering (uGPS) and parsing (NeoGPS).

Well, I certainly appreciate your taking the time to check it out and offer some feedback.

Cheers,
/dev

jboyton

  Serial.print("Memory used: "); Serial.println(2048 - freeMemory());

This is executed after a sentence is decoded, so I'm not sure what it's measuring.  And it turns out that string literals use RAM.  For a pleasant surprise, I would suggest replacing all the string literals with F("string literal"), at least the ones that are printed.  Boy, this is a topic in itself.
Using F() reduces the RAM usage equally (by 110 bytes) between the two sketches. So the difference in RAM usage -- which is what I was trying to measure -- remains the same.

It's true that exactly where you measure the free RAM matters. I wanted to do it in the context of a sketch since that's what counts, ultimately. If I write a memory efficient library that requires lots of external memory then it's a false economy. Likewise with processing speed. Of course the example matters and I chose one that is closer to my needs. A different one could make your approach shine.


It really gets down to whether you need the information from the device, or just the bytes.  If you are comparing values or calculating a distance, you need the numbers.  This is the NeoGPS et al niche.

If you are just logging or displaying the bytes, perhaps in a different order, you only need... bytes.  A much simpler problem.
Yes, that's the big difference. It's one reason I didn't just use TinyGPS. All I want, with a few exceptions, is to either display the data or write it as text to an SD card. All of the number crunching I plan to do later on a machine that actually has a DIV instruction and a data path wider than 8 bits.


> ...by calling the parse function from the ISR...

Hey!  That's exactly the async mode I mentioned in the README.  I usually work in the Cosa environment, and it's very easy to hook `decode` to the ISR.  You have to be careful because the fix data is volatile when it's populated by an ISR.
So it *is* in there? I'll have to take a look. I combined my software serial and gps code because I couldn't figure out the syntax that C++ wanted in order to set up a call from one library to another. I even had an example to work with but I kept getting errors. I'm a C++ moron. So I just glued my two libraries together and planned to deal with it later.


> ...with interrupts re-enabled.

:o   You sir, are a square peg.  :D
Unorthodox perhaps, but I like the fact that in my main loop I can do things that take tens of milliseconds without either breaking them up into multiple steps or using a large RX buffer.

I might be able to parse with interrupts still disabled but I haven't bothered to figure out what the longest path is through my parse function. If that path exceeds the worst case available time between RX interrupts it wouldn't work. At 38400 baud, after accounting for time in interrupts, I think I have something less than 15 microseconds worst case.


Well, I certainly appreciate your taking the time to check it out and offer some feedback.
No worries, it was fun. I may have use for your library in the future.

Cheers.

-dev

This still doesn't make any sense:

>> NeoGPS: sketch = 9950; SRAM = 515
>> my GPS: sketch = 6532; SRAM = 457

> the difference in RAM usage... remains the same.


What the heck is going on?

<google, look at assembly, insert freeRam code, test a few things...>

What the heck is going on?  I don't have any statics or string literals, so where is it coming from? >:(

Time.  It's in the Time library.  statics galore, about 100 bytes.  What a waste!  I spend all that effort reducing RAM, and then reuse a fairly standard library... which is not stingy with RAM.  Doh!

Fine.  Out it goes...

> the syntax that C++ wanted in order to set up a call from one library to another.

If you have defined the gps object in your .INO like this

  NMEAGPS gps;

You would probably need this in the ISR.cpp file:

  #include "NMEAGPS.h"
  extern NMEAGPS gps;


The #include tells the compiler about the type NMEAGPS, and the extern tells the compiler that the instance gps is (for real) somewhere else.  Both are needed to use it from the ISR like this:

gps.decode( byte );

You must be familiar with the volatile issue, though.  The ISR will update the members of gps.fix() at any time, so you can't go grabbing them whenever you want.  If you really want to try this, I would suggest looking at the sister project of NeoGPS, CosaGPS.  The example CosaNMEAdevice.INO shows how to safely use the fix while the ISR is doing things in the background.  I'm not sure it's worth your time, though.

I'm gonna go ditch that Time library now... grrr.  Again, thanks for the feedback, as this was a real oversight on my part.

Cheers,
/dev

-dev

After ditching the library, I'm getting the same numbers.  Apparently the unused statics in that Time library were optimized out.  I'll have to keep looking, I guess.

Cheers,
/dev

jboyton

#22
Jan 07, 2015, 08:20 pm Last Edit: Jan 07, 2015, 08:22 pm by jboyton
Thanks for the suggestions with regard to calling a method in another class. I didn't keep that code and don't remember what the error messages were but it  wasn't due to a lack of an include file or an extern. I was originally trying this with two classes in the same sketch.

SdFat was what I was attempting to use as an example. That library provides a hook for setting the file creation/modification date and time. You pass SdFat a pointer to your function that provides the datetime (from RTC or GPS). Then when SdFat needs the datetime it calls your function. I thought I could do something similar: pass a pointer to the gps parse method to my softserial class. If the pointer wasn't null then the ISR would call it.

But no matter what I did I got compiler errors. I just didn't know what I was doing.

I noticed that in your library nothing is declared volatile. Isn't the rule that if it's accessed both within and outside an ISR the variable must be declared volatile? How do you guarantee that the latest value obtained from within an ISR isn't held in a register rather than being written to memory?

jboyton

After ditching the library, I'm getting the same numbers.
Numbers from what? What are you comparing?

jboyton

The difference between the two sketches was 58 bytes (515-457), not 100. The soft serial library has a 64 byte buffer whereas my uGPS library (which has an integrated software serial) has no buffer. There were a few other small savings when I put those two libraries together. I think that accounts for most of the difference.

Just to see, I made a third sketch using the original, pre-merge, zGps and zSoftSerial libraries. The totals were:

111 object size
6736 sketch size
532 RAM used

That makes it 17 bytes larger than the Neo sketch and is perhaps a better apples to apples, zGps vs NeoGps, comparison.

I really wanted to compare the whole bundle, as I'm using it, but short of actually integrating NeoGps into my project.

By the way, is it possible to print gps.fix().dateTime? The compiler threw an error when I tried it with Serial.print().

-dev

> pass a pointer to the gps parse method

Ah, yes.  This kind of pointer is called a "pointer to member", and is a fairly obscure part of C++.

In the old days, the C++ "compiler" was really just a preprocessor... it translated C++ to C, then invoked the C compiler on the "translation."  From that, I learned that C++ code

  obj.method( arg )

is really

  obj1a_method( &obj, arg )

in C.  Using obj1a_method as a function pointer is fairly common in C.  But in C++, having a pointer to a "method" is not enough.  You still need an obj to be passed in as the implicit first argument.  The syntax for declaring, initializing and using a pointer-to-method if pretty noisy, if you ask me.

> I noticed that in your library nothing is declared volatile.

Yes, that's because (1) it does need it internally, even if it executes in an interrupt context, and (2) most people will not be using in such a context.  If it were used in an ISR, the entire object would be considered volatile:

  volatile NMEAGPS gps;

Although technically correct, the techniques for safely accessing the object obviate the need for the keyword.  You still have to

  • wait until a sentence is completed (which spans multiple interrupts, and is a topic in itself), then
  • outside the ISR, disable interrupts and get what you need (quickly!), whether that's one value, or a safe_copy of gps.fix().
  • Finally, reenable interrupts and use the value or safe_copy at your leisure.
The Cosa example uses this to coordinate waiting for a completed sentence:

  volatile bool frame_received;

The ISR sets it, and non-ISR code watches for it.  Without the volatile keyword, the non-ISR code could, as you said, be checking a cached value, not the variable.  Or the compiler could even optimize it away.

> [getting the same] Numbers from what?

Before and after switching the Time libraries.  It had no effect.

> the difference was 58 bytes (515-457), not 100.

Yes, but the 515-457 includes the difference between the NeoGPS and uGPS objects too.  I think you have to add (86-43) to those 58 to compare the non-object SRAM numbers.  That's 101-byte difference in the rest of the code, which I thought was just NeoGPS.

> The soft serial library has a 64 byte buffer...

Ah, I see.  I was sure flummoxed.

> 111 object size, 6736 sketch size, 532 RAM used
>
> That makes it 17 bytes larger than NeoGPS


See, I look at that and think (111-43) = 68 bytes difference in the object size, so where did I use (68-17) = 51 more bytes?  GAH!   :smiley-cry:

I'm ok, really.  :)  I think you're saying it's in some other place.

> is it possible to print gps.fix().dateTime?

Yes, but not via Serial.print.  Instead, do this:

  Serial << gps.fix().dateTime;

This is really a C++ convention for "streaming" something out.  I've never really liked it, but here we are...  This statement actually calls a function whose name is operator <<.  Yes, the left-shift operator is overloaded with new behavior.  Streamers.h and .cpp provide it thus:

  extern Stream & operator <<( Stream & outs, const tmElements_t & t );

Here you can see how Serial is the first argument, and the dateTime is the second.  It returns a Stream so that you can chain a bunch of these together:

  Serial << F("dt = ") << gps.fix().dateTime << '\n';

There are other operator << routines that have a second argument of FlashString and char; they have the same name, but different arguments.  That's overloading.  It kind of resolves like this:

Code: [Select]
  (Serial << F("dt = ")) << gps.fix().dateTime << '\n'; // outputs PROGMEM "dt = ", which

  (Serial) << gps.fix().dateTime << '\n';               //  returns Serial. Now we have

  Serial << gps.fix().dateTime << '\n';                 //  Evaluate again...

  (Serial << gps.fix().dateTime) << '\n';               // outputs formatted dateTime, which

  (Serial) << '\n';                                     // returns Serial.  Next we have

  Serial << '\n';                                       // which outputs the newline

  (Serial);                                             // returns Serial.  Return value not used.

You can always grab the code from Streamers.cpp (which uses <<) and make your own routine with the individual prints.  It ends up being the same amount of code.

Thanks for the extra information.  I think I'll rest easier!

Cheers,
/dev

jboyton

Thanks for the C++ lesson. I'm familiar with the concepts from having taken a short course but I never used any of what little I learned. So I'm far from fluent. I know it's going to take effort to become better at it.

I wouldn't sweat the SRAM bytes. It's almost certainly an artifact of how I set up my test.

-dev

I just came across an interesting post about the awful state of the NMEA "standard"... from five years ago!  It's called Why GPSes suck, and what to do about it.  It was written by the leader of the gpsd group.  gpsd is a GPS daemon for UNIX.  The article recounts all the reasons why parsing is so messed up.  He ends with this:

Quote
Expecting GPS-aware applications to keep track of all this stuff would be just nuts. The best way to cope is to have a dedicated service layer that specializes in knowing about GPS idiosyncracies, hides all that ugliness, and presents a simple Time-Position-Velocity-reporting interface (TPV) to the application layer above.
This really captures the state of affairs quite nicely, and it enumerates many of the reasons I started on NeoGPS in the first place.  Frugal RAM usage is not one of his reasons, though, so most of the gpsd code is not suitable for the embedded environment.  It does have some concepts that I need to review.

Oh, well... it's worth a read if you're in to this sort of thing. :smiley-confuse:

Cheers,
/dev

jboyton

I just came across an interesting post about the awful state of the NMEA "standard"... from five years ago!  It's called Why GPSes suck, and what to do about it.
An entertaining rant to read, thanks for that. It doesn't make life easier for the poor guy who just wants to set his clock made with an Arduino via GPS but at least it provides some comic relief.

rockeronline00

What about SPI for communication between GPS and Arduino?   SoftwareSerial is too slow
MS in Computer Science
Drones addicted
Musician and composer

Go Up