Squeezing code into UNO to drive Adafruit Ultimate GPS Logger / microSD

[ninja2]

Over some years I've developed a big sketch to monitor and display things like motion and electrics on an electric scooter. It works real well on a Mega with an Adafruit Ultimate GPS Logger shield logging time, position, speed and much more to SD. The whole sketch compiles to about 65,000 bytes.

Recently I've been trying to do a cut down version for UNO but even after cutting out as much as I can it's a squeeze. Oh and I can't get it to work yet, the SD card simply fails to open the file on the microSD

Here's the routine that writes a header into the CSV file. It only fails on UNO. The bits commented out just to reduce sketch size all work on a MEGA, or if the GPS code and activity is disabled:

void logHeader(){                                        // write header rows to CSV file
  dataString = CJ_ID; 
  //String columnHeadings = "";
  //columnHeadings = "#,millis,MM,DD,hh,mm,ss,Day,Lat,Long,kph,course,ALTg,ALTb,Amps,Volts,Watts,C_Ard,C-Motor,C-Brake,kJrun,kJchg,gap,Mode,Hits";


  File dataFile = SD.open(logFileName, FILE_WRITE);
  if (dataFile) {dataFile.println(dataString);               // if the file is available, write to it:
                 Serial << "  Hdr:" << dataString  << "\n";  // and report
                 //dataFile.println(columnHeadings);           // write the row of column headings
                 }
  else          {Serial << "Hdr OPEN FAIL [" << logFileName << "]\n";
                 SD_OK = false;}
  dataFile.close();
  }

One possibility is it's still simply too big for UNO, but I'm not sure. Here's the compile data for my most stripped down version:

Sketch uses 24,524 bytes (76%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,748 bytes (85%) of dynamic memory, leaving 300 bytes for local variables. Maximum is 2,048 bytes.

I added a Check_mem() routine that reports free memory of only around 100 bytes, and stack pointer minus heap pointer of only around 200 bytes. I'm not sure if these are too low / tight.

If I comment out all the GPS functionality the SD works, but I can't get them working together. And it's not hardware as I've tried on 2 UNOs and 2 GPS shields, same results. Obviously the sketch is much smaller when GPS commented out so that suggests it's a size problem.

But I think there may be another possible cause: a timer/counter or interrupt conflict between the various libraries. I am using the libraries:

/* -- Included libraries ------------------------------------------------------------ */
#include <SoftwareSerial.h>
#include <Adafruit_GPS.h>
Adafruit_GPS GPS(&GPSport);                             // initalise GPS object
#include <SPI.h>                                        // for microSD comms
#include <SD.h>                                         // microSD library
#include <Streaming.h>                                  // for simplified serial string commands
#include <TimedAction.h>                                // for loop timing

I have used derived GPS handling from the Adafruit "Parsing" example which depends on an "Output Compare" interrupt from Timer/Counter 0.

The whole sketch is spread over 8 tabs so too big to post inline. But here is a dropbox link to the sketch in a ZIP for those who are keen

I'm hoping someone will point me to a reference guide somewhere describing which timer/counters are used in background by the common Arduino libraries and functions (like Serial.print), for UNO and MEGA.

Any and all suggestions welcomed! TIA

[dev_1]

I'm hoping someone will point me to a reference guide somewhere describing which timer/counters are used in background by the common Arduino libraries and functions (like Serial.print), for UNO and MEGA.

Well, you can always look at the source code:

    <arduino-install-dir>/hardware/arduino/cores/arduino

... or someplace similar, depending on what version of the IDE you're using.

I would reconsider using the Adafruit_GPS library. It is very inefficient. It uses a timer interrupt to read characters that were received during the RX interrupt. This causes more interrupt processing, and means that every character is copied multiple times, in and out of the serial input buffer... into the Adafruit double-buffered sentence array, where it gets searched multiple times with strcmp and strchr. Ugh.

Try my library, NeoGPS. It is smaller, faster, and very configurable (see Performance, RAM and Program Size comparison pages). The most basic example, NMEA.ino, can be easily modified to stream to the SD card, in a CSV format that is very similar to what you show above. See Streamers.cpp. There is an advanced example for logging to an SD card at 10Hz.

And I hope you're not using the String class. You can save 1600 bytes of program space by using char arrays instead, not to mention untold hours of frustration when it starts to act weird. Having only 200 bytes of free space is definitely a problem, and truly precludes using String.

BTW, most of us will not download from dropbox. Just attach the files to your post. Use the "Attachments and other options" in the lower-left corner of the post editor.

Cheers,
/dev

[ninja2]

great info /dev, thanks.

Up until recently I was using TinyGPS and I only recently changed to the Adafruit library because TinyGPS would not report the time from the battery-backed RTC in the GPS until it had a fix. That's something I definitely need!

I will certainly have a look at NeoGPS - do you think it will report RTC time before the GPS gets its fix? Or could you easily make it do that?

I've attached ZIP as you suggested, and yes I must plead guilty of heavy use of String. I take your advice but have to say I was relieved when String came about as char manipulation was a frustrating challenge for my mediocre coding skills. If you look at the SD tab in my code you'll see where I use String to build up 'datastring' for writing a row to the CSV. It would be great to know how you would do this using char arrays?

cheers / TIA

AdaSDlog.zip (9.16 KB)

[dev_1]

do you think it will report RTC time before the GPS gets its fix?

Yes, NeoGPS reports all the fields that are sent by the GPS device. If it has an internal RTC, those values will be reported.

NeoGPS also tells you which fields are not available, with the valid flags. For example, when the GPS does not have a fix and is using the internal RTC instead, NeoGPS will report the following values:

gps_fix fix; // a fix structure defined by the NeoGPS library
fix = gps.read(); // read the next available fix structure from the NeoGPS library

if (fix.valid.status) {  // true
  datafile.print( fix.status ); // 0 is STATUS_NONE
}

if (fix.valid.dateTime) { // true because RTC used to set field values
  datafile.print( fix.dateTime.month );
  datafile.print( ',' );
  datafile.print( fix.dateTime.date );
  datafile.print( ',' );
  datafile.print( fix.dateTime.month );
  datafile.print( ',' );
  datafile.print( fix.dateTime.hours );
  datafile.print( ',' );
  datafile.print( fix.dateTime.minutes );
  datafile.print( ',' );
  datafile.print( fix.dateTime.seconds );
  datafile.print( ',' );
  fix.dateTime.set_day(); // This expensive day-of-week calculation must be called explicitly. See Time.H
  datafile.print( fix.dateTime.day );
  datafile.print( ',' );
} else {
  datafile.print( F(",,,,,,,") ); // nuthin' available!
}

if (fix.valid.location) {  // false because no fix yet that will set lat/lon
  datafile.print( fix.latitude(), 10,6 ); // -89.123456
  datafile.print( ',' );
  datafile.print( fix.longitude(), 11,6 ); // -179.123456
  datafile.print( ',' );
} else {
  datafile.print( F(",,") ); // nuthin' available!
}

etc...

It would be great to know how you would writing a row to the CSV using char arrays?

As above... write the pieces and you don't even need a char array!

Cheers,
/dev

[ninja2]

OK good, I'll look into this.

BTW are you at the Lat/Long in your example? Like, near the South Pole !? :o

[dev_1]

LOL, no... that was just to show you how many digits you can have for a lat/lon.

I thought the 2nd argument was the size of the output buffer (4th argument), but it is actually the minimum width of the conversion. The docs simply say "you are responsible for providing enough space." The valBuffer array has 15 bytes, so that's 14 plus NUL-termination, which is fine. Your original format is correct:

   datafile.print( fix.latitude(), 7,6 ); // -89.123456

I usually use the integer forms because they have 10 significant digits (float has ~7), and it reduces the program size by several K if you don't have to link in the floating-point library. Here's a routine that prints a long integer as if it were a higher-precision float (no such thing on the Arduino):

void printL( Print & outs, int32_t degE7 )
{
  // Extract and print negative sign
  if (degE7 < 0) {
    degE7 = -degE7;
    outs.print( '-' );
  }

  // Whole degrees
  int32_t deg = degE7 / 10000000L;
  outs.print( deg );
  outs.print( '.' );

  // Get fractional degrees
  degE7 -= deg*10000000L;

  // Print leading zeroes, if needed
  int32_t factor = 1000000L;
  while ((degE7 < factor) && (factor > 1L)){
    outs.print( '0' );
    factor /= 10L;
  }
  
  // Print fractional degrees
  outs.print( degE7 );
}

To print the lat or the lon, do this:

   printL( datafile, fix.latitudeL() ); // prints int like a float
   datatfile.print( ',' );

That's an excerpt from NMEAloc.ino.

Thanks for asking!
/dev

[ninja2]

Cool stuff, more ways to cut program size. I can see how size reduction could become an obsession

you obviously know C/C++ very well ... a bit off topic but I came across this snippet of code in the LowLatencyLogger example in the SdFat library distribution:

// Acquire a data record.
void acquireData(data_t* data) {
  data->time = micros();
  for (int i = 0; i < ADC_DIM; i++) {
    data->adc[i] = analogRead(i);
  }
}

// Print a data record.
void printData(Print* pr, data_t* data) {
  pr->print(data->time);
  for (int i = 0; i < ADC_DIM; i++) {
    pr->write(',');
    pr->print(data->adc[i]);
  }
  pr->println();
}

I've never seen the "->" operator before and haven't been able to find usefull help via google or arduino ref. Any clues? It reads as "minus, greater than" to me !?

Also in this snippet of yours:

datafile.print( F(",,,,,,,") ); // nuthin' available!

I gather you like the "F" syntax for any literal strings. I think this means the string is stored in code space rather than RAM, have I got that right?

[dev_1]

I've never seen the "->" operator before...

That's called the "arrow" or "dereferencing" operator. Some good answers here. When you want a certain member of a structure, you use

  myStruct.member = 17;

But if you have a pointer to a structure (i.e., the address of the structure), then to get a certain member, you have to "derefence" the pointer-to-struct:

void foo(  myStructType *myStructPtr )
{
  myStructPtr->member = 17;
}

When the argument to a function is a structure, it is more efficient to pass in the address of the structure, rather than copying the structure onto the stack for the function. (Modern compilers do "other" things than that, but let's suspend our disbelief for now...).

I much prefer passing a C++ "reference" instead of a "pointer", which uses the ampersand instead of the asterisk:

void foo( myStructType & myStructRef )
{
  myStructRef.member = 17;
}

...because it implemented with the efficiency of pass-by-address, but has the clean "." usage inside the function.

I gather you like the "F" syntax for any literal strings.

Not really, but...

this means the string is stored in code space rather than RAM

Yes, that's why you use it. You almost always run out of RAM before you run out of program space. It's an artifact of the 8-bit AVR MCU Harvard Architecture.

Cheers,
/dev

[ninja2]

cool, it's time I got into structures, pointers and referencing ...
so if I've understood the Stackoverflow answers:

void foo(myStructType *myStructPtr )
{
  myStructPtr->member = 17;
}

is also equivalent to:

void foo(myStructType *myStructPtr )
{
  (*myStructPtr).member = 17;
}

When using * (and maybe & too) it doesn't seem to matter if its justified to the left, middle or right?
By this I mean all three below seem to mean the same thing:

void foo(myStructType* myStructPtr )
void foo(myStructType * myStructPtr )
void foo(myStructType *myStructPtr )

Meanwhile after removing 'String datastring' and using .print(F( to the max my SD logger sketch seems to be quite stable now and works on the UNO. I'm still going to try NeoGPS and using longs in place of floats as you've mentioned.

On that note, I can't get my head around the meaning of "Print" in line 1 of your example:

void printL( Print & outs, int32_t degE7 )}

So outs is a structure but what does Print mean here, perhaps it's a reserved word for actually printing to the standard output?

And all the above questions are taken to the extreme (from my perspective) in the LowLatencyLogger snippet again:

// Print a data record.
void printData(Print* pr, data_t* data) {
  pr->print(data->time);
  for (int i = 0; i < ADC_DIM; i++) {
    pr->write(',');
    pr->print(data->adc[i]);
  }
  pr->println();
}

"pr->print(data->time);" whoa :o

[dev_1]

all three below seem to mean the same thing:

void foo(myStructType* myStructPtr )

void foo(myStructType * myStructPtr )
void foo(myStructType *myStructPtr )

Yes, "identifiers" like "foo" or "myStructPtr" are a sequence of characters from the set { 0, 1, 2, ... 9, A, B, C, ... Z, _, a, b, c, ... z }, but the first character of an identifier can't be a digit. Identifiers are delimited by, well, non-identifier characters, like space or asterisk. So this would also work:

     void foo(myStructType*myStruct)

Not pretty, but the compiler figures out what you mean. Spaces and newlines are free, and they don't make your program run any slower. They just make it easier to read your code. Increasing readability is perhaps the best thing you can do to improve your code.

I can't get my head around the meaning of "Print" in line 1 of your example:

void printL( Print & outs, int32_t degE7 )

In a function declaration, the arguments are always specified as a argument type and an argument name (an identifier). In this case, "Print &" must be the type, because "outs" is the argument name. "outs" can be used inside "printL", and the compiler makes sure that whatever the caller passed in gets "substituted" during execution. In this case, Serial is passed in for "outs", and so this code

    outs.print( '-' );

...actually executes as if it were written like this:

    Serial.print( '-' );

Now that you know "Print" is a type name, you can go looking for it. It turns out that it is part of the Arduino "core". All the source code for the pre-defined Arduino bits was installed with the IDE. In this case, you will find out that Print is a C++ class name (nearly the same as a struct), defined in these files:

    <install-dir>\hardware\arduino\avr\cores\arduino\Print.h
    <install-dir>\hardware\arduino\avr\cores\arduino\Print.cpp

WinGrep is a decent tool for searching through all that source code. If you're on Linux, grep is the classic search utility. If you're on a Mac, you're on yer own.

The Arduino Reference pages are silent about the Print class, except for one small mention here. Basically, the Print class is the generic declaration of something that can print chars, bytes, ints, etc. types. Stream adds the generic "reading" concept to generic "printing". Things like Serial and datafile are derived from Stream, so they can be used to print or read chars, bytes, ints, etc.

This is really what it takes to learn the Arduino environment: what software (classes, functions) is available and how do you control the hardware (MCU registers and bit definitions). Most people learn by looking at examples that are close to what they want. I would not suggest reading all the source code in the install directory. Example programs will lead you down that rabbit hole often enough.

// Print a data record.

void printData(Print* pr, data_t* data) {
  pr->print(data->time);
  for (int i = 0; i < ADC_DIM; i++) {
    pr->write(',');
    pr->print(data->adc[i]);
  }
  pr->println();
}

"pr->print(data->time);" whoa :o

Apparently the data_t struct has as member called time (probably an integer type). Since the data argument is passed in is a pointer to the structure, not the structure, you have to use the arrow operator, not the dot. I would suggest this is more readable:

// Print a data record.
void printData( Print & pr, data_t & data)
{
  pr.print( data.time );
  for (int i = 0; i < ADC_DIM; i++) {
    pr.write( ',' );
    pr.print( data.adc[i] );
  }
  pr.println();
}

Oh well.

Another hurdle is coding for the embedded environment. Lots of folks come from the desktop environment, where you have a nice multi-tasking, virtual memory OS. Not so here, so String and its use of dynamic memory is generally taboo.

Cheers,
/dev

[ninja2]

excellent /dev, thanks again.

"If you're on Linux, grep is the classic search utility. If you're on a Mac, you're on yer own. :)"

and being on a winPC doesn't even get a mention oh well.

I'm very comfortable with hardware in general including MCU's, with their limited memory, registers, bit mapping, tinterrupts, timers etc and assembler code, but other than my many arduino sketches I've not delved into higher level programming language like C/C++. But it's (slowly) proving a useful adventure. Even windows search finds specific text within a directory tree these days, so I'll have a look around.

I agree you're rewrite of printData is easier to read ... maybe you're starting a movement?

many thanks for yr generous answers...

cheers

[ninja2]

OK I got NMEA.ino working on my UNO with Adafruit U/GPS logger shield no problems. I copied the example to my NeoNMEA sketch attached as a zip.

You'll see I've imposed all my styles preferences, sorry. best you simply ignore those oh, and I edited all the text down to just the parts I needed, and I broke the sketch up across a few Tabs ... as I do for all my sketches. I find this is best way for me to learn what someone else's code does... or at least what I think it does!

I have a bunch of questions but I'll focus on the big ones first ...

1st Question: My whole reason for heading down into this “rabbit warren” was simply to improve my primary sketch "SCOO6" by adding the ability to read RTC time from GPS whether or not a fix was available. Plus there’s a constant side goal of improving overall reliability of the code (no processing stumbles) and reducing its size (code and RAM). Plus this is my idea of fun.

Now I've invested heaps into SCOO6 over a few years. In summary it is fitted on an electric scooter and reads several sensors, as well as the GPS data then computes all the values, formats and records them to microSD (approx. 4 times/sec = every 250ms). Plus it formats and prints the status and data to the IDE monitor (or DEBUG_PORT as you call it), and displays a subset as text on a small dash-mounted LCD.

It uses a crude “pseudo real time multi-tasking” system I’ve built up using the TimedAction library, with four timed loops running (asynchronously) at 250ms, 500ms, 2s and 5s. (my AdaSDlog sketch has the same mechanism but only at 250ms and 2s loops, see ZIP attached to my earlier post).

So now I see NMEA.ino in action I’m wondering how much control I’ve got over NeoGPS at the sketch level, so I can understand how to merge NeoGPS functionality into SCOO6 without a massive rewrite (by my standards :)). For example your comment before the “doSomeWork” function says it’s called every sec (approx) during the GPS quiet time. What if I want to do things faster (or slower) ? Can I control the frequency? How would I control CSV headings written to SD and add non-GPS data columns?

2nd question: when I tried to move the doSomeWork function:

static void doSomeWork(){trace_all(DEBUG_PORT,gps,fix_data);}

to the last Tab it would not compile and gave an error messages starting with:

AAsetup:1: error: previous declaration of 'void setup()' with 'C++' linkage

So I re-instated it back so it is located before ‘void setup()’ in the definitions (1st) tab, and it compiles fine. WTF?

I’m probably being lazy not looking at your other examples yet. They may answer some of these questions, but I’m punting you can get me started quicker

NeoNMEA.a.zip (2.87 KB)

[dev_1]

being on a winPC doesn't even get a mention

WinGrep is for Windows.

your comment before the "doSomeWork" function says it's called every sec (approx) during the GPS quiet time. What if I want to do things faster (or slower) ? Can I control the frequency?

Well, the while loop is just skipped if no characters are available from gps_port, so you can just call all the TimedAction__.check()'s before or after the while loop (or the separate GPSloop()):

   GPSloop();
    Timed0250Action.check(); // every 250 ms
  //Timed0500Action.check(); // every 500 ms
  //Timed1000Action.check(); // every 1 sec
    Timed2000Action.check(); // every 2 sec

This will work fine for many programs. Specifically, as long as none of the checks take longer than ~64ms, you will reliably get all GPS data.

Unfortunately, it is very common for SD writes to occasionally take ~100ms. When that happens, the GPS device can send about 100 characters. But the gps_port input buffer is only 64 characters, so the last 36 characters are dropped. When the SD write completes and it gets back to GPSloop, it reads the first part of a GPS sentence and never sees the last part. That sentence is incomplete and ignored. I am describing this problem again.

So you can do other things as often as you like, faster or slower than 1Hz. Just make sure to call GPSloop within 64ms, or GPS data will be lost. Really, GPSloop is like "GPScheck". Or you can decide it doesn't matter if you lose one update every now and then. AdaFruit_GPS sidesteps this problem by using a timer interrupt to read from the serial port into its own buffer. Expensive. You're not ready for it yet, but there is an example that shows how to avoid input buffer overflow during an SD write.

How would I control CSV headings written to SD and add non-GPS data columns?

Modify the example Streamers.cpp? That file will build with any configuration, so you'll see lots of "#ifdef" wrappers around print statements. If a particular option is disabled, it won't try to print that piece of data. For example, if you don't enable the GPS altitude field, this code:

    #ifdef GPS_FIX_ALTITUDE
      if (fix.valid.altitude)
        outs.print( fix.altitude(), 2 );
      outs << ',';
    #endif

...is skipped by the compiler. It won't be in your uploaded program at all. And there are two major sections in there, one for floating-point output and one for integer output. That choice is controlled by the USE_FLOAT define at the top of the file. Comment it out for integer output, or uncomment it for floating-point output.

Or you can just copy and paste the pieces you want into your own files. You know how to write a header, and you know how to write the pieces. Streamers is just an example of printing all the pieces that are available, as described here. It also shows how to check the valid flags before using a fix member. NMEAloc.ino is a simple example that shows how to print just the location, without using Streamers.

when I tried to move the doSomeWork function to the last Tab it would not compile and gave an error message:

    AAsetup:1: error: previous declaration of 'void setup()' with 'C++' linkage

Yeah, the Arduino IDE tries to "help" beginners by making a few (incorrect) guesses, especially with INO files. At this stage of the game, I do not recommend separating loop and setup into separate files. You've never seen that have you? I cannot find the magic combination to get keep the IDE from "helping", so just "stay within the lines" and you'll be ok.

If you really want to have separate files, you have to use .CPP files. And you'll need a header file (i.e., .H file) for each of those .CPP files. And then you need to declare them extern in the H file, and include the headers where the extern declarations are needed. See attached. :-/

BTW, static functions cannot be referenced across files. They can only be used in the CPP where they are defined, and there's no reason to put them in the H file. Most software geeks would say that's good for decoupling and name collisions, but the Arduino gods aren't exactly software geeks.

Cheers,
/dev

NeoNMEAv2.zip (1.62 KB)

[ninja2]

After reading your detailed post and looking through NeoNMEAv2 I had 14 questions/issues written down. I’ll try to be brief to avoid overload:

I reckon your 1ms/GPS char estimate is based on GPS at 9600baud and 10 bytes/char. I read somewhere that NeoGPS can run up to 38,400baud. I was planning to upgrade to that rate once I had the basics going. That should cut processing down from ~64 to ~16ms, right?

As you suggest I could run GPScheck (i.e. GPSloop) several times in main loop between each call to TimedAction.check().This is how TinyGPS operates too, but to avoid the whole dropping chars issue I do think a well written interrupt driven solution would be more reliable/elegant/pleasing. IMO a well written interrupt should be efficient not ‘expensive’. The Adafruit implementation, despite its shortcomings, does work on Mega, and UNO OK. I must look into your NMEA_isr example and NMEAGPS_INTERRUPT_PROCESSING !

Do I really need to explicitly #include Arduino.h? Isn’t this always included anyway by the IDE? (When I comment it out everything still compiles and works?)

Are the object size reports printed during setup in bytes?

I can see the trace_header() function inside Streamers.cpp but what does it do? As an aside, is there a way to test if the Serial monitor is present? If so I would use it to turn off all serial output unless it’s actually present.

And why Serial.flush()? The Serial.flush() reference says “Waits for the transmission of outgoing serial data to complete.” But it also says it inherits from the Stream class and that Stream.flush() reference says “clears the buffer once all outgoing characters have been sent.” The latter makes more sense to me, but I’ve never used this function before now so wondering what it adds/fixes/avoids. Perhaps it's just a precaution, to guarantee the GPS "pipeline" is truly empty before proceeding into loop()?

I studied your use of Tabs with underlying .cpp and .h files. I understand this is probably the correct/better approach but I’m reluctant to change to that right now, as I’m very used to the format of standard Tabs each producing an .ino file and I’ve got ‘bigger fish to fry’ at present. Also I fixed the compile error by just adding protoype functions for doSomeWork() and GPScheck() to the main tab and my sketch compiles fine. Prototypes have been required since IDE 1.6.6 (for multiple tabs) which is a bit annoying but I’ve got used to it in all my recent sketches.

I’m heading off explore Streamers.cpp … standby !

cheers + thanks

[ninja2]

PS: wondering why this edit to Streamers.cpp won't compile:

#ifdef GPS_FIX_ALTITUDE
      if (fix.valid.altitude)
        //outs << fix.altitude_cm();
        outs << fix.alt.whole(); // doesn’t work, even with () removed too
      outs << ',';
    #endif

neither does this:

    #ifdef GPS_FIX_SPEED
      if (fix.valid.speed)
        //outs << fix.speed_mkn(); // integer knots * 1000
        outs << fix.spd.whole;     // integer knots
      outs << ',';
    #endif

[dev_1]

ninja2:
I reckon your 1ms/GPS char estimate is based on GPS at 9600baud and 10 bytes/char.

Yes.

I read somewhere that NeoGPS can run up to 38,400baud. I was planning to upgrade to that rate once I had the basics going.

NeoGPS itself does not have any limitation on speed. For the typical configuration (i.e. Nominal), NeoGPS can process GPS data at ~825000 baud, much faster than you'll probably need. I think you're referring to NeoSWSerial, a more-efficient alternative to SoftwareSerial, and a derivative of a library created by jboyton. Because it is based on the millis() timer and Pin Change interrupts, it can only run up to 38400.

If pins 8 & 9 are available, I strongly recommend AltSoftSerial, which is even more efficient, and can run up to 115200.

That should cut processing down from ~64 to ~16ms, right?

Yes, but that is the maximum polling time. If any other part of your program takes longer than 16ms, you will lose GPS characters because the input buffer overflows in an even shorter time. But I see you are considering the interrupt-driven approach, which obviates the whole overflow issue.

...I do think a well written interrupt driven solution would be more reliable/elegant/pleasing. IMO a well written interrupt should be efficient not ‘expensive’. The Adafruit implementation, despite its shortcomings, does work on Mega, and UNO OK. I must look into your NMEA_isr example and NMEAGPS_INTERRUPT_PROCESSING !

LOL, I never said it didn't work. I just said "Ugh!" Having twice the interrupts will affect the reliability of other, time-critical parts of the system, like a software serial library. The Adafruit_GPS ignores the checksum, so I'm not sure it's really working OK. It just doesn't tell you something is broken.

Do I really need to explicitly #include Arduino.h?

Nah, it's probably left-over from some legacy code or a CPP file. I haven't tried it recently, but I think you had to put it in CPP files if no other H file included it.

Are the object size reports printed during setup in bytes?

Yes, sizeof is in bytes.

I can see the trace_header() function inside Streamers.cpp but what does it do?

It just prints two PROGMEM strings: the concatenated names of the configured fields in the gps_fix structure, and the concatenated names of configured fields in the NMEAGPS class.

The language allows you to break double-quoted strings into two adjacent pieces. For example, these two statements will print the same thing:

Serial.print( "string literals can be concatenated or one big literal" );
Serial.print( "string literals can be" " concatenated"
                  " or "
                  "one big literal" );

So Streamers.cpp uses this technique to allow pieces of the header line to be conditionally compiled:

const char gps_fix_header[] __PROGMEM =
  "Status,"

  #ifdef GPS_FIX_LOCATION
    "Lat,Lon,"
  #endif

I skipped over the messy [date, time, date/time] choice so you can see that the "Lat,Lon," portion of the header string is conditional with the GPS_FIX_LOCATION configuration item.

As an aside, is there a way to test if the Serial monitor is present? If so I would use it to turn off all serial output unless it’s actually present.

No, but you could disable output until a character/command is received from Serial. Output a message like "Press any key for debug" during setup, then press ENTER in the Serial Monitor window to turn on the debug prints.

And why Serial.flush()?

It's not really required unless you're timing something. When a software serial port is used, I like to start loop without having a bunch of characters queued up for output. Then if the received GPS data is ok for a bit, then starts to have CS errors, I can guess that there's "too much" activity. When setup prints more than 64 characters, it will have been blocking anyway, until the last characters could be queued up. So it's not just the GPS pipeline, it's to help understand the MCU interrupt load.

I understand this is probably the correct/better approach but I’m reluctant to change

Okey doke.

Also I fixed the compile error by just adding protoype functions for doSomeWork() and GPScheck() to the main tab

Ugh! (That's not to say it doesn't work... ) It's really silly to have to declare a static function in a different file from where it is defined. That's contrary to the intent of those "local" functions. They will be visible in all the INOs. Double-plus Ugh.

wondering why this edit to Streamers.cpp won't compile:

        outs << fix.spd.whole;     // integer knots

Because the Print class is not fully implemented. :-/ The whole member is of type int16_t, a 16-bit signed integer. Print implements the streaming "operator <<" with a 16-bit unsigned integer only, so the compiler complains that nobody provided "operator <<" with a 16-bit signed integer. This will work:

        outs.print( fix.spd.whole );     // integer knots

...because the Print class provides a print method that takes a 16-bit signed integer.

Hmm... I just noticed some recent changes to NeoSWSerial have reduced its reliability. I'll have to investigate...

Cheers,
/dev

[dev_1]

I just rolled back those NeoSWSerial changes. You might want re-download the ZIP and re-copy the NeoSWSerial.H and CPP files to the Arduino libraries directory.

Cheers,
/dev

[ninja2]

OK will do.

I've enjoyed last couple of days working though (almost) all of your NeoGPS examples. A "million" questions came up along the way, fortunately most were answered in a subsequent examples. Here's a few bullet points from my journey:

Of the all sketches I tried NMEAloc was the only one I couldn't get to work. But I didn't spend much time on it as it didn't seem as important or interesting as NMEA_isr and NMEASDlog, both of which I got working today. It's notable that my variant of NMEASDlog uses less than 19,000 bytes of code space and 970 bytes in SRAM, compared to my old AdaSDlog / Adafruit GPS library sketch at around 25,000+250 bytes. This (plus moving to SdFat) provided stability on the UNO and my variants of NMEASDlog are working on UNO and MEGA. yay!

But I do have one annoying issue - see my last post (#10) in this thread about SD / SdFat compatibility. I thought I had fixed it by adding serial.flush() after every log event to the SD (in my old AdaSDlog sketch). But it (SD file system corruption) happened again running NMEASDlog a little while ago. As u know that uses logfile.flush(); so now I'm stumped again. Any thoughts? As it doesn't happen every time I'm wondering if it's just bad timing, turning off power just when SD is being written? What is best way to kill the sketch when it's logging? In my most developed sketch I use a pin change toggle interrupt to turn logging on or off. And I rarely get file corruption that way.

I accidentally tried to run NMEAorder at 5Hz but it wouldn't work until I reverted to 1Hz.

I'm picking up some neat tricks (assuming you're a good influence ) AFAIK the two statements

for (;;);                 // WTF:)
while(true){}             // ok ok ok ok ok ...

both do the same thing: stop the program in it's footsteps, or at least send it into an infinite loop. right?

I only spotted one (very) unimportant error in NMEASDlog at line 408:

digitalWrite(6,HIGH); //change pin 6 to LED
// digitalWrite(LED,HIGH);// like this

Looking at the way you use code and the way you structured NeoNMEAv2 has really been a good insight into some C/C++ basics. I've gather the Arduino IDE is designed to hide the whole cpp and h file system, which makes sense for 90% of the forum enthusiasts and wannabe's. Dividing a sketch up into several tabs only inserts artificial divisions between tabs that don't influence the compiler. A sketch with 5 tabs = 5 .ino files, looks like one big .ino to the IDE compiler. Tabs are only to provide a more horizontal structure, something I prefer to really long vertical sketches. Better suits my two-dimensional mind

'static' and 'extern' can be ignored in a .ino and so I've never needed to know about them, but diving into NeoNMEAv2 showed me (something about) how they work and it helps having to tweak the various NeoGPS config files too.

BTW you may be pleased to know I've started using Notepad++. Editing NMEA_cfg.h and similar files by renaming them to .rtf and opening in MS Word (don't laugh) was a hassle.

A friend gave me an old book on C++ a while back, by Brian Overland circa 2001. I'll dive into that soon + occasionally.

Off to start merging the best of the above into my big Scooter instrumentation sketch. Shouldn't take more than a month or two

cheers

[dev_1]

NMEAloc was the only one I couldn't get to work.

Did it build? Was there no output? It could have been the configuration, too.

I accidentally tried to run NMEAorder at 5Hz but it wouldn't work until I reverted to 1Hz.

Yes, it definitely assumes 1Hz to detect the 1-second intervals.

It's notable that my variant of NMEASDlog uses less than 19,000 bytes of code space and 970 bytes in SRAM, compared to my old AdaSDlog / Adafruit GPS library sketch at around 25,000+250 bytes.

Hmm, I don't think the old AdaSDlog uses only 250 bytes of RAM. Maybe it has 250 bytes of RAM free? When I built AdaSDlog for an UNO, I got 24148+1677. Could you post your NeoGPS version for comparison, please?

But I do have one annoying issue - SD file system corruption happens... Any thoughts?

Does it happen with the unmodified NMEASDlog.ino? It's ok to print some GPS commands at the beginning, or change the baud rates, but don't modify anything else. Attach it and the cfg files for me to review. Different SD cards may be more or less susceptible to the corruption, because I've never seen it. One other fellow got empty files, but no corruption.

I'm wondering if it's just bad timing... What is best way to kill the sketch when it's logging?

Flushing the SD file is equivalent to closing it, at least in the last version of SDFat that I looked at. Just pick some external event and do a flush, and stop logging for some time (or forever) so that the card can be extracted, or the Arduino can be powered off. And provide some indication that logging has stopped. Turn an LED on or off? Blink it?

In my most developed sketch I use a pin change toggle interrupt to turn logging on or off. And I rarely get file corruption that way.

I can't see your code, so I hope the PCInt doesn't do anything except set a variable (flag) that is checked in the main loop.

I only spotted one (very) unimportant error in NMEASDlog at line 408:

digitalWrite(6,HIGH); //change pin 6 to LED

// digitalWrite(LED,HIGH);// like this

I was making a suggestion to the original author that he use a named constant instead of a "6" so that the reader would know what was connected to pin 6.

I'm picking up some neat tricks (assuming you're a good influence )

You know what they say about ASSume, right? Yes, those are two ways to "hang" a program.

Editing NMEA_cfg.h and similar files by renaming them to .rtf and opening in MS Word (don't laugh) was a hassle.

I'm crying on the inside.

The Overland book is decent, and it was written before the language got bloated. Much clearer for beginners. Stay away from malloc/free, new/delete, try/catch and the String class, as they are not suitable for the embedded environment (Arduino vs. PC). They may not be in your edition.

Cheers,
/dev

[ninja2]

/dev:
Did it build? Was there no output? It could have been the configuration, too.

yes it did build, and start. I'll revisit it later and see if I can get it going

/dev:
Hmmm, I don't think the old AdaSDlog uses only 250 bytes of RAM. Maybe it has 250 bytes of RAM free?

Sorry it was getting late when I wrote that, I should have said free RAM. Here's the compile message for my NeoSDlog variant (not SIMULATE_SD mode). So it's all good news,

Sketch uses 18,444 bytes (57%) of program storage space. Maximum is 32,256 bytes.
Global variables use 1,082 bytes (52%) of dynamic memory, leaving 966 bytes for local variables. Maximum is 2,048 bytes.

/dev:
When I built AdaSDlog for an UNO, I got 24148+1677. Could you post your NeoGPS version for comparison, please?

The compile numbers above for NeoSDlog are almost identical to NMEASDlog because so far I've only been restructuring your examples to my pinouts plus my preferred styles/layout. So no substantial changes...

/dev:
Does it happen with the unmodified NMEASDlog.ino?

Maybe, but I haven't run it enough I've probably only run it once. But again my NeoSDlog is essentially same function.

But I think you've answered my question anyway. I reckon that because NMEASDlog/NeoSDlog just keep logging ad-infinitum, there is no guaranteed safe way to stop the program logging and remove the SD without risking corruption.

/dev:
I hope the PCInt doesn't do anything except set a variable (flag) that is checked in the main loop.

Yes sets a flag, as shown below. But this is in my big (and reliable) SCOO6 sketch that requires a MEGA and is still based on TinyGPS (not for long!). Again I haven't done any substantial mods to my Neo sketches.

void toggleLogging(){                                    // Interrupt Service Routine (int0 = pin 2)
  if ((millis() - lastDebounceTime) > debounceDelay){    // only register if debounce period exceeded
    logButtonPressed = true; 
    lastDebounceTime = millis();}
  }

I will consider adding something like this to NeoSDlog

/dev:
The Overland book is decent ...

Good to know, thanks again.

stand by ...