GPS NMEA data logging to SD card. Help needed

I finally got a chance to work with this and work through some bugs, some minor some not so much. ... And therefore none of my mapping programs can read it, it just is "Waiting for fix"

I read this as: I changed my code. I'm not going to show it to you. I still need help.

Did I mis-read something?

Yes, code please. Code makes all the difference. I can tell you why the comma is missing, though:

        } else if (gps_count == 5) {       <-- not 6, because...
          // check the sentence type
          *gps_line++ = c;
          gps_count++;                     <-- ...this increments to 6

          if (strncmp( start_of_gps, rmc, 6 ) == 0) {
            got_rmc  = true;
            gps_line = &rmc[6];
          } else if (strncmp( start_of_gps, gga, 6 ) == 0)) { <-- not start_of_rmc
            got_gga  = true;
            gps_line = &gga[6];
          } else
            gps_state = WAITING_FOR_START; // ignore the rest

This had the effect of saving the comma in start_of_gps, not in rmc or gga. Mea culpa. :blush: The same thing is in check_sdd:

        } else if (sdd_count == 5) {

Sounds like you found all the missing things I warned you about. :slight_smile:

Cheers,
/dev

Thanks Devin I'll give that a shot, that's the only problem I have left that I was stuck on. I have some little things yet like wait to fix before writing to SD card but I don't foresee any problems coming up with that myself. Writing my own functions on the other hand ...... I know my way around several languages just enough to hack together some "boilerplate" stuff just from being around it for 30+ years

Other than the missing comma delimiter the output is spot on perfect with 1 sec intervals and zero data loss. In fact judging from some quick the timing measurements made on a better serial terminal (Actually on a Nexus 7 that I use to troubleshoot RS-232 and RS-485 based industrial/commercial gear) I should have enough quiet time to easily be able to grab a couple of more sentences, for instance I forgot about needing the SDMTW which is water temperature output to my main computer and GPGSA has a couple of more DOP (dilution of precision) values some GIS scripts I have can use to sanity check and parse out bad or less accurate data based on those values. (and a big reason why I wanted to keep everything in straight NMEA format) I also still want to implement Bluetooth or basically a choice between Serial0 (USB) and Serial3 (BT) output implemented with a SPDT switch. (I want this to be essentially stand alone) Then I'll have something like no one else has and you just can't buy from the store. (Always a top design goal for all my personal gear)

When I make Public the final code I want to give you credit for the functions because without them this is just another poorly implemented NMEA multiplexer. (And I've tried at least half a dozen and wrote one myself) Credit to SlashDevin and a link back to your github page, or do you have some other preference?

Thanks

Netbook2Charplotter Project Lost Grove Lake Mapping Project

That did the trick, it works for ‘live’ mapping, the SD card data imports correctly and I even ran it through a parser that checks GPS time code … A little over 10 minute test 100% 1 second intervals, zero data loss

Here is the corrected code for this function in case anyone else wants to give it a shot

//----------------

enum NMEA_state_t { WAITING_FOR_START, RECEIVING };

static NMEA_state_t gps_state, sdd_state;

static char start_of_gps[8];  // temporary place until first 6 chars received
static char start_of_sdd[8];

static char rmc[120] = "$GPRMC";
static char gga[120] = "$GPGGA";
static char dpt[120] = "$SDDPT";

static bool got_rmc;
static bool got_gga;
static bool got_dpt;

static char *gps_line;
static uint8_t gps_count;

static uint32_t last_rx = 0UL; //  to remember when we last received a char

//............

static void check_gps()
{
  bool got_something = false;

  while (Serial1.available()) {
    c = Serial1.read();
    got_something = true;

    switch (gps_state) {
      case WAITING_FOR_START:
        if (c == '

) {
          // Start of sentence!
          gps_count = 0;
          gps_line  = &start_of_gps[0];

*gps_line++ = c;
          gps_count++;

gps_state = RECEIVING;
        }
        break;

case RECEIVING:
        if (c == ‘\r’) {
          *gps_line++ = c;
          gps_count++;

} else if (c == ‘\n’) {
          *gps_line++ = c;
          gps_count++;
          *gps_line++ = 0; // NUL-terminate

gps_state = WAITING_FOR_START; // get next sentence

} else if (gps_count == 5) {
          // check the sentence type
          *gps_line++ = c;
          gps_count++;

if (strncmp( start_of_gps, rmc, 6 ) == 0) {
            got_rmc  = true;
            gps_line = &rmc[6];
          } else if (strncmp( start_of_gps, gga, 6 ) == 0) {
            got_gga  = true;
            gps_line = &gga[6];
          } else
            gps_state = WAITING_FOR_START; // ignore the rest

} else if (gps_count < sizeof(rmc) - 3) {
          // save chars if there’s room
          *gps_line++ = c;
          gps_count++;
        }
        break;
    }
  }

if (got_something)
    last_rx = millis();

else if (millis() - last_rx > 10UL) {

if (gps_state == WAITING_FOR_START) {

// Quiet time, write the GPS sentences…

if (got_rmc) {
        sd_file.write( rmc );
        Serial.write( rmc );
        got_rmc = false;
      }

if (got_gga) {
        sd_file.write( gga );
        Serial.write( gga );
        got_gga = false;
      }

// … and the SDDPT sentence, when/if it shows up during the quiet time.

if (got_dpt) {
        sd_file.write( dpt );
        Serial.write( dpt );
        got_dpt = false;
      }
    }
  }
}

//…

static char  *sdd_line;
static uint8_t sdd_count;

static void check_sdd()
{
  while (Serial2.available()) {
    c = Serial2.read();

switch (sdd_state) {
      case WAITING_FOR_START:
        if (c == ’


) {
          // Start of sentence!
          sdd_count = 0;
          sdd_line  = &start_of_sdd[0];

          *sdd_line++ = c;
          sdd_count++;

          sdd_state = RECEIVING;
        }
        break;

      case RECEIVING:
        if (c == '\r') {
          *sdd_line++ = c;
          sdd_count++;

        } else if (c == '\n') {
          // SDDPT complete

          *sdd_line++ = c;
          sdd_count++;
          *sdd_line = 0; // NUL-terminate

          got_dpt = true;

          sdd_state = WAITING_FOR_START;

        } else if (sdd_count == 5) {
          // check the sentence type
          *sdd_line++ = c;
          sdd_count++;

          if (strncmp( start_of_sdd, dpt, 6 ) == 0) {
            sdd_line = &dpt[6];
            got_dpt  = false;
          } else
            sdd_state = WAITING_FOR_START; // ignore the rest

        } else if (sdd_count < sizeof(dpt) - 3) {
          // save chars if there's room
          *sdd_line++ = c;
          sdd_count++;
        }
        break;
    }
  }
}

That did the trick… zero data loss.

Well, there ya’ go! Did you test with the sounder, too? The reason I ask is because you mentioned something in your previous post:

I forgot about needing the SDMTW

You may have noticed that the code in check_sdd is slightly different. This is because:

(1) saving the SDDPT may be happening at a different time from the GPS sentences. For example, the sounder busy time may be happening during the GPS quiet time, and vice versa;
(2) there was only one sentence to watch for, SDDPT; and
(3) got_dpt indicates when the sentence is completely received, while got_rmc and got_gga indicate when those sentences start.

Regarding (1), I am a little concerned that you could lose sounder data while the logging is happening in check_gps, during the GPS quiet time. If SDMTW and SDDPT only come once a second, and they are very short (< 64 chars total), no data should be lost.

Regarding (2) and (3), adding the SDMTW requires reconciling those flags, which means a little renaming and some extra flags. I would suggest a new flag, getting_xxx, to indicate that a sentence is RECEIVING, while the old got_xxx indicates that a complete sentence has been received and can be logged:

//----------------

enum NMEA_state_t { WAITING_FOR_START, RECEIVING };

static NMEA_state_t gps_state, sdd_state;

static char start_of_gps[8];  // temporary place until first 6 chars received
static char start_of_sdd[8];

static char rmc[120] = "$GPRMC";
static char gga[120] = "$GPGGA";
static char dpt[60] = "$SDDPT";
static char mtw[60] = "$SDMTW";

static bool got_rmc, getting_rmc;
static bool got_gga, getting_gga;
static bool got_dpt, getting_dpt;
static bool got_mtw, getting_mtw;

static char *gps_line;
static uint8_t gps_count;

static uint32_t last_rx = 0UL; //  to remember when we last received a char

//............

static void check_gps()
{
  bool got_something = false;

  while (Serial1.available()) {
    c = Serial1.read();
    got_something = true;

    if (c == '

This also makes the two routines more similar, which helps with readability. It may also help consolidate things into common routines, a process called “refactoring.” As you cut and paste in new sentence types to be saved, or other devices to be monitored, the code gets longer and harder to read, maintain and debug. Yes, there might be a few things missing. :slight_smile:

I also changed the switch statements to if statements that force starting a new sentence whenever the ‘$’ is received. This will make it more immune to lost data: if the LF is dropped, the next sentence will start correctly. Similarly, if it has been 50ms since any data was received (in the quiet time) and we’re still in the RECEIVING state, force a reset to WAITING. We must have missed the end-of-sentence.

Since you took some timing data with the Nexus 7, I know you’re aware that increasing the number of sentences you log decreases the quiet time window, and thus decreases how much time you’ve got to log and forward those sentences. There are other techniques for interleaving the reading and writing, but the code is a little more complicated.

I also want to implement… a choice between Serial0 (USB) and Serial3 (BT) output implemented with a SPDT switch.

Yes, very doable. You will declare something like this at the top:

Stream *output_port = &Serial;

Then in setup, you can check the state of the switch:

  if (digitalRead( pin ) == HIGH) {
    Serial3.begin( 19200 ); // BT setup?
    output_port = &Serial3;
  } else {
    Serial.begin(115200);  // Serial setup?
  }

… and replace all Serial.write with output_port->write:

      if (got_gga) {
        sd_file.write( gga );
        output_port->write( gga );    <--- changed
        got_gga = false;
      }

Easy, no? You could even make the change now, without the setup changes, as output_port defaults to Serial.

Credit to SlashDevin and a link back to your github page, or do you have some other preference?

Gawrsh! Karma’s fine! Wait, is cash an option? :wink:

Cheers,
/dev) {
     // Start of sentence!
     gps_count = 0;
     gps_line  = &start_of_gps[0];

*gps_line++ = c;
     gps_count++;

getting_gga = false;
     getting_rmc = false;

gps_state = RECEIVING;

} else if (gps_state == RECEIVING) {

if (c == ‘\r’) {
       *gps_line++ = c;
       gps_count++;

} else if (c == ‘\n’) {
       *gps_line++ = c;
       gps_count++;
       *gps_line++ = 0; // NUL-terminate

if (getting_gga) {
         getting_gga = false;
         got_gga     = true;

} else if (getting_rmc) {
         getting_rmc = false;
         got_rmc     = true;
       }

gps_state = WAITING_FOR_START; // get next sentence

} else if (gps_count == 5) {
       // check the sentence type
       *gps_line++ = c;
       gps_count++;

if (strncmp( start_of_gps, rmc, 6 ) == 0) {
         getting_rmc = true;
         got_rmc     = false;
         gps_line = &rmc[6];
       } else if (strncmp( start_of_gps, gga, 6 ) == 0) {
         getting_gga = true;
         got_gga     = false;
         gps_line = &gga[6];
       } else
         gps_state = WAITING_FOR_START; // ignore the rest

} else if (gps_count < sizeof(rmc) - 3) {
       // save chars if there’s room
       *gps_line++ = c;
       gps_count++;
     }
   }
 }

uint32_t now = millis(); // just call this once

if (got_something)
   last_rx = now;

else if (now - last_rx > 10UL) {

// Quiet time!
   
   if (now - last_rx > 50UL) {
     //  It’s been too long, give up on what we were RECEIVING
     gps_state == WAITING_FOR_START;
   }

if (gps_state == WAITING_FOR_START) {

// Write the saved sentences…

if (got_rmc) {
       sd_file.write( rmc );
       Serial.write( rmc );
       got_rmc = false;
     }

if (got_gga) {
       sd_file.write( gga );
       Serial.write( gga );
       got_gga = false;
     }

if (got_dpt) {
       sd_file.write( dpt );
       Serial.write( dpt );
       got_dpt = false;
     }

if (got_mtw) {
       sd_file.write( mtw );
       Serial.write( mtw );
       got_mtw = false;
     }

}
 }
}

//…

static char   *sdd_line;
static uint8_t sdd_count;

static void check_sdd()
{
 while (Serial2.available()) {
   c = Serial2.read();

if (c == ’


This also makes the two routines more similar, which helps with readability. It may also help consolidate things into common routines, a process called "refactoring." As you cut and paste in new sentence types to be saved, or other devices to be monitored, the code gets longer and harder to read, maintain and debug. Yes, there might be a few things missing. :)

I also changed the `§_DISCOURSE_HOISTED_CODE_8_§` statements to `§_DISCOURSE_HOISTED_CODE_9_§` statements that force starting a new sentence whenever the '$' is received. This will make it more immune to lost data: if the LF is dropped, the next sentence will start correctly. Similarly, if it has been 50ms since any data was received (in the quiet time) and we're still in the RECEIVING state, force a reset to WAITING. We must have missed the end-of-sentence.

Since you took some timing data with the Nexus 7, I know you're aware that increasing the number of sentences you log decreases the quiet time window, and thus decreases how much time you've got to log and forward those sentences. There are other techniques for interleaving the reading and writing, but the code is a little more complicated.

> I also want to implement... a choice between Serial0 (USB) and Serial3 (BT) output implemented with a SPDT switch.

Yes, very doable. You will declare something like this at the top:

§DISCOURSE_HOISTED_CODE_10§


Then in `§_DISCOURSE_HOISTED_CODE_11_§`, you can check the state of the switch:

§DISCOURSE_HOISTED_CODE_12§


... and replace all `§_DISCOURSE_HOISTED_CODE_13_§` with `§_DISCOURSE_HOISTED_CODE_14_§`:

§DISCOURSE_HOISTED_CODE_15§


Easy, no? You could even make the change now, without the `§_DISCOURSE_HOISTED_CODE_16_§` changes, as `§_DISCOURSE_HOISTED_CODE_17_§` defaults to `§_DISCOURSE_HOISTED_CODE_18_§`.

> Credit to SlashDevin and a link back to your github page, or do you have some other preference?

Gawrsh! Karma's fine! Wait, is cash an option? ;)

Cheers,
/dev) {
      // Start of sentence!
      sdd_count = 0;
      sdd_line  = &start_of_sdd[0];

      *sdd_line++ = c;
      sdd_count++;

      getting_dpt = false;
      getting_mtw = false;

      sdd_state = RECEIVING;

    } else if (sdd_state == RECEIVING) {

      if (c == '\r') {
        *sdd_line++ = c;
        sdd_count++;

      } else if (c == '\n') {
        // Sentence complete

        *sdd_line++ = c;
        sdd_count++;
        *sdd_line = 0; // NUL-terminate

        if (getting_dpt) {
          getting_dpt = false;
          got_dpt     = true;

        } else if (getting_mtw) {
          getting_mtw = false;
          got_mtw     = true;
        }

        sdd_state = WAITING_FOR_START;

      } else if (sdd_count == 5) {
        // check the sentence type
        *sdd_line++ = c;
        sdd_count++;

        if (strncmp( start_of_sdd, dpt, 6 ) == 0) {
          sdd_line    = &dpt[6];
          getting_dpt = true;
          got_dpt     = false;

        } else if (strncmp( start_of_sdd, mtw, 6 ) == 0) {
          sdd_line    = &mtw[6];
          getting_mtw = true;
          got_mtw     = false;

        } else
          sdd_state = WAITING_FOR_START; // ignore the rest

      } else if (sdd_count < sizeof(dpt) - 3) {
        // save chars if there's room
        *sdd_line++ = c;
        sdd_count++;
      }
    }
  }
}

This also makes the two routines more similar, which helps with readability. It may also help consolidate things into common routines, a process called “refactoring.” As you cut and paste in new sentence types to be saved, or other devices to be monitored, the code gets longer and harder to read, maintain and debug. Yes, there might be a few things missing. :slight_smile:

I also changed the §_DISCOURSE_HOISTED_CODE_8_§ statements to §_DISCOURSE_HOISTED_CODE_9_§ statements that force starting a new sentence whenever the ‘$’ is received. This will make it more immune to lost data: if the LF is dropped, the next sentence will start correctly. Similarly, if it has been 50ms since any data was received (in the quiet time) and we’re still in the RECEIVING state, force a reset to WAITING. We must have missed the end-of-sentence.

Since you took some timing data with the Nexus 7, I know you’re aware that increasing the number of sentences you log decreases the quiet time window, and thus decreases how much time you’ve got to log and forward those sentences. There are other techniques for interleaving the reading and writing, but the code is a little more complicated.

I also want to implement… a choice between Serial0 (USB) and Serial3 (BT) output implemented with a SPDT switch.

Yes, very doable. You will declare something like this at the top:

§_DISCOURSE_HOISTED_CODE_10_§

Then in §_DISCOURSE_HOISTED_CODE_11_§, you can check the state of the switch:

§_DISCOURSE_HOISTED_CODE_12_§

… and replace all §_DISCOURSE_HOISTED_CODE_13_§ with §_DISCOURSE_HOISTED_CODE_14_§:

§_DISCOURSE_HOISTED_CODE_15_§

Easy, no? You could even make the change now, without the §_DISCOURSE_HOISTED_CODE_16_§ changes, as §_DISCOURSE_HOISTED_CODE_17_§ defaults to §_DISCOURSE_HOISTED_CODE_18_§.

Credit to SlashDevin and a link back to your github page, or do you have some other preference?

Gawrsh! Karma’s fine! Wait, is cash an option? :wink:

Cheers,
/dev

Actually I have not only extended it to get the SDMTW sentence from the sounder I have also extended it to get the GPGSA sentence from the GPS. Although these can be written to the SD card too I commented out those lines and only write it to Serial0 (USB). Although I experienced no data loss with them writing to the SD card there was a noticeable difference in the deadtime between writes to the serial port.

Regardless of whether they are on or off I am seeing perfectly ‘framed’ and timed data coming out the USB port where with other NMEA combiners it’s a bit of a mish mash of frame times and timing which still works because the speed is relatively slow but isn’t exactly right. Perfectly framed and timed serial data is simply more reliable and your code makes that happen

Anyways here is your code extended to grab the GPGSA and SDMTW sentences and send them out the USB port (Or to the SD card if you uncomment a couple of lines)

//----------------

enum NMEA_state_t { WAITING_FOR_START, RECEIVING };

static NMEA_state_t gps_state, sdd_state;

static char start_of_gps[8];  // temporary place until first 6 chars received
static char start_of_sdd[8];

static char rmc[120] = "$GPRMC";
static char gga[120] = "$GPGGA";
static char gsa[120] = "$GPGSA";
static char dpt[120] = "$SDDPT";
static char mtw[120] = "$SDMTW";

static bool got_rmc;
static bool got_gga;
static bool got_gsa;
static bool got_dpt;
static bool got_mtw;

static char *gps_line;
static uint8_t gps_count;

static char  *sdd_line;
static uint8_t sdd_count;

static uint32_t last_rx = 0UL; //  to remember when we last received a char

//............

static void check_gps()
{
  bool got_something = false;

  while (Serial1.available()) {
    c = Serial1.read();
    got_something = true;

    switch (gps_state) {
      case WAITING_FOR_START:
        if (c == '

) {
          // Start of sentence!
          gps_count = 0;
          gps_line  = &start_of_gps[0];

*gps_line++ = c;
          gps_count++;

gps_state = RECEIVING;
        }
        break;

case RECEIVING:
        if (c == ‘\r’) {
          *gps_line++ = c;
          gps_count++;

} else if (c == ‘\n’) {
          *gps_line++ = c;
          gps_count++;
          *gps_line++ = 0; // NUL-terminate

gps_state = WAITING_FOR_START; // get next sentence

} else if (gps_count == 5) {
          // check the sentence type
          *gps_line++ = c;
          gps_count++;

if (strncmp( start_of_gps, rmc, 6 ) == 0) {
            got_rmc  = true;
            gps_line = &rmc[6];
          } else if (strncmp( start_of_gps, gga, 6 ) == 0) {
            got_gga  = true;
            gps_line = &gga[6];
          } else if (strncmp( start_of_gps, gsa, 6 ) == 0) {
            got_gsa = true;
            gps_line = &gsa[6];  
          } else
            gps_state = WAITING_FOR_START; // ignore the rest

} else if (gps_count < sizeof(rmc) - 3) {
          // save chars if there’s room
          *gps_line++ = c;
          gps_count++;
        }
        break;
    }
  }

if (got_something)
    last_rx = millis();

else if (millis() - last_rx > 10UL) {

if (gps_state == WAITING_FOR_START) {

// Quiet time, write the GPS sentences…

if (got_rmc) {
        sd_file.write( rmc );
        Serial.write( rmc );
        got_rmc = false;
      }

if (got_gga) {
        sd_file.write( gga );
        Serial.write( gga );
        got_gga = false;
      }
      
      if (got_gsa)  {
        //sd_file.write( gsa );
        Serial.write( gsa );
        got_gsa = false;
      }

// … and the SDDPT sentence, when/if it shows up during the quiet time.

if (got_dpt) {
        sd_file.write( dpt );
        Serial.write( dpt );
        got_dpt = false;
      }
      
      if (got_mtw) {
        //sd_file.write( mtw );
        Serial.write( mtw );
        got_mtw = false;
      }
    }
  }
}

//…

//static char   *sdd_line;
//static uint8_t sdd_count;

static void check_sdd()
{
  bool got_something = false;
  
  while (Serial2.available()) {
    c = Serial2.read();
    got_something = true;
    
    switch (sdd_state) {
      case WAITING_FOR_START:
        if (c == ’


) {
          // Start of sentence!
          sdd_count = 0;
          sdd_line  = &start_of_sdd[0];

          *sdd_line++ = c;
          sdd_count++;

          sdd_state = RECEIVING;
        }
        break;

      case RECEIVING:
        if (c == '\r') {
          *sdd_line++ = c;
          sdd_count++;

        } else if (c == '\n') {
          *sdd_line++ = c;
          sdd_count++;
          *sdd_line = 0; // NUL-terminate

         sdd_state = WAITING_FOR_START;

        } else if (sdd_count == 5) {
          // check the sentence type
          *sdd_line++ = c;
          sdd_count++;

          if (strncmp( start_of_sdd, dpt, 6 ) == 0) {
            got_dpt  = true;
            sdd_line = &dpt[6];            
          } else if (strncmp( start_of_sdd, mtw, 6 ) == 0) {
             got_mtw = true;
             sdd_line = &mtw[6];
          } else  
            sdd_state = WAITING_FOR_START; // ignore the rest

        } else if (sdd_count < sizeof(dpt) - 3) {
          // save chars if there's room
          *sdd_line++ = c;
          sdd_count++;
        }
        break;
    }
  }
}

Is the sounder somehow synchronized with GPS, or does it just output SDDPT & SDMTW once per second?

They don’t really specify but it’s just a once per second and my secondary finder which I’m using for testing is fixed at 4800 and has a fixed output of SDDPT, SDDBT and SDMTW … SDDPT is SDDBT (Depth below transducer) plus the transducer offset (In my case 0.8ft) for the main rear transducer, and about the same for the front trolling motor mounted transducer. (Final offset yet to be determined and will be done in the field) The offset is set in the finder configuration and figured internally to get SDDPT … My main finder is a Lowrance and I have control over sentences and the baud rate although I’ll set it for 4800 to match the Garmin which is fixed.

I should explain my setup … I have a Lowrance Elite 7x HDI mounted on the console (16ft heavy duty MonArk jon boat decked out) which sets in the middle of the boat with the transducer in the rear. In front with the trolling motor is a Garmin 300C. The NMEA serial outputs of each are terminated in a 2 conductor plug like you see for a microphone on a radio. In order to make calibration much much easier I use a ‘puck’ type active patch antenna placed right over the transducer as close as possible in the front and back. This NMEA combiner/logger will have a matching 2 conductor jack for the NMEA serial and a SMA jack for the antenna. I just plug in the antenna NMEA serial I want to use, normally the Console finder and rear antenna and if I need to navigate from the trolling motor in close quarters I just power down the logger and swap out the NMEA serial and antenna connectors and start it back up. If you have a standard setup you have the transducer in the rear and the GPS antenna in the middle so you have a 2+ meter difference you have to compensate for

That sounds like a nice setup, and that you’ve put some serious thought into it!

The reason I asked about the sounder synchronization is to see if there’s a bug lurking in the current code. If the sounder emits its sentences during the GPS sentences, here’s what I think is happening:

Sounder synched.jpg

All the sentences get accumulated correctly, and the got_xxx flags are all true when the GPS quiet time comes around. The Serial.write has plenty of time to complete, and so does the sd_file.write (not shown). This is perfect!

However, if the sounder emits its sentences during the GPS quiet time, the current set of got_xxx flags will be set too soon (e.g., after the first character of DPT). Because it’s during the quiet time, the got_dpt flag will be detected immediately by check_gps, and the Serial.write will write the previous sentence’s data and clear the flag. Then, the rest of the DPT will be received and saved. The same thing happens for the MTW. Here’s a timing diagram for this behavior:

Sounder not synched.jpg

Notice how the got_dpt and got_mtw flags are true for just one loop() iteration – they’re just little spikes. Also notice that the Serial.write happens in two batches, one for the GPS sentences (when the quiet time starts), and one later for the sounder sentences (in the middle of the quiet time). It’s even possible for the second Serial.write to extend into the next GPS second, during the next GGA. I don’t think any data will be lost because the sounder sentences are so short. (Again, sd_file.write not shown.)

The effect is that previous second’s depth is written after the current second’s location. It’s not a huge difference in terms of the whole lake, but I wanted to make you aware of this time offset. To eliminate this offset, you need to check a flag that’s only valid at the end of the sentence, not the beginning. One way to do that is have one set of flags for the beginning of the sentence (i.e., getting_xxx), and a different set of flags for the end of the sentence (i.e., got_xxx). That’s in my previous code post.

Of course, waiting for the end of the sentence will also push it closer to the end of the GPS quiet time. :slight_smile: If the quiet time ends before the end of the sentence is received, it will just be saved until the next quiet time comes around. This would be a little closer to the “nearest” location sample (i.e., nearest in time), but maybe no significant difference in the long run.

If you really want one burst per second, regardless of where the sounder data appears in the GPS second, I’ll have to think a little more. Right now, the Serial.write timings are dependent on when the unsynchronized sounder data is emitted WRT the GPS second. I would even say it’s “susceptible” (to random offsets) and a little “incoherent” (previous data timestamped in current data).

Well, as long as you’re aware of what’s going on, you can make an informed decision.

Sounds like you’re ready to go!

Cheers,
/dev

Actually that's not unusual behavior and the way these depth mapping programs process the data is it gets the GPS sentence and then gets the last known depth reading. So it kind of backs up and they do that for a reason and it's got something to do with what you are talking about and is just some sort of work around for the limitations of the NMEA 0183 standard since the 'talkers' aren't synced up like with the more advanced NMEA 2000 standard. I guess everything averages out, at least close enough for this application.

I'll know next week when I get it out on the water and run some calibration and accuracy tests in an area I have well mapped and know real well. I really don't foresee any problems, everything is testing well on the bench and none of my mapping and GIS software is having any problem with it. It seems more solid than the old way where I fed the GPS to the software in one COM port and the sounder in another COM port

I ran it for over an hour and a half logging to the SD card and also logging the USB output and ran it through a parsing scripts that looks for missing seconds in the GPS time data and it was again 100% and as far as I can tell no dropped data on either output. I finally found a use for a couple of brand new in the package 256MB SD cards I have. They are too small for about everything else but this is using less than 1 MB over one and a half hours so these are a perfect fit. My intention is to take it out after every trip and put a fresh one in. Basically I'll save that as a daily backup

Thanks for the help, I'll come back and post a link when I post a write up of this project on my blog. It should be easy to copy even if you don't know a lot about Arduino and have some basic soldering skills and easily modified by someone with an intermediate knowledge. Common parts and a popular platform and easy to reproduce ... That's a change because most of my stuff isn't easily reproducible (Like turning a 2nd generation netbook into a tablet computer in early 2010)

Something’s been nagging at my mind about the unsynchronized SDMTW…

If the SDMTW sentence is half-received when the GPS quiet time starts, then your current code will save an invalid SDMTW in which the first half consists of new bytes (say 30 bytes), and the second half is old bytes from the previous SDMTW (say another 30 bytes). Here’s the timing diagram:

Sounder Timing 2.jpg

Note how at the end of the Serial.write (or sd_file.write), check_gps sees that got_mtw is true, and it writes the mtw array.

Unfortunately, check_sdd has received only the first half of the new SDMTW sentence. The second half of the mtw array contains bytes that were saved from the previous SDMTW.

This leads to two things:
(1) the values are goofy, because a temp may start out with ‘5’ from a new ‘59.4’ reading (half-received), but it ends with ‘0.1’ from a previous reading of ‘60.1’, which makes it look like ‘50.1’.
(2) since you are saving the checksum, these half-new/half-old sentences will be rejected.

The same thing could happen to the SDDPT.

You will only see this when the sounder sentences are being received when the GPS quiet time begins, so I don’t think you’ve seen this bug manifest yet. I would guess that the odds are somewhere between 1:5 and 1:20 that the timing for a session is bad. Once it’s going, the timing drift should be fairly low. It’s really a power-on condition, I think.

At a minimum, I would recommend that you guard the writing of the sounder sentences until sdd_state is WAITING, by adding one if statement in check_gps:

      // ... and the SDDPT sentence, when/if it shows up during the quiet time.

      if (sdd_state == WAITING_FOR_START) {   <--- add this line...

        if (got_dpt) {
          sd_file.write( dpt );
          Serial.write( dpt );
          got_dpt = false;
        }
      
        if (got_mtw) {
          //sd_file.write( mtw );
          Serial.write( mtw );
          got_mtw = false;
        }
      }                                       <--- ... and this line

This extra if statement will keep it from writing while it’s still RECEIVING. When the sounder sentences are completely received (and the sdd_state goes back to WAITING), then they will be written.

I’d hate for you to get out there and save a bunch of useless sounder data, just because it happened to emit its sentences when the GPS quiet time starts. Maybe cycling the power on the sounder would “throw the dice” and change the timing? Fortunately, this one if statement will always work, even if the timing drifts in and out of the bad window during a session.

Cheers,
/dev

Thanks I'll give that a try when I get it back together. I'm permanently wiring everything and in fact am waiting for the last part of the case to print out so I can call it a night. Nice thing about it is from here it's all firmware. I won't even have to wait until I get home I can just upload adjusted firmware from my Lenovo Tablet (Let's hear it for full sized USB ports)

If anyone is interested the code can be had on my GitHib repository and the final results are documented on my blog

Very nice graphics! At first, I thought you scanned an old paper topo map, with wrinkles... Then I realized it you had merged the LIDAR into the topo. :-[

Is the overlay the results of your scan? As far as I can see, they match the elevation lines perfectly. The lake hasn't been filled very long, so that's what you expected, right?

BTW, I notice that you have an sd_file.flush() in loop(). Did you find that it was necessary? I would have expected sd_file.write to flush when necessary (1 or 2 SD write cycles per second). This could cause 2 extra SD write cycles during the quiet time, if the sounder data arrives in that interval. Not a big deal, unless it starts creep into the next GPS second.

Well, I'm glad it worked for you! It must be a relief to get through it.

Cheers, /dev

Those are just base maps I downloaded from the Iowa Geographic Map Server in 1000x1000 meter squares as GeoTIFF's and stitched together. The lake overlay was made by tracing the topo map at the designed full lake level. Some changes were made (like roads and a bridge removed) since the topo was made in the 1990's so it's a best guess estimate.

The overlay thing is something I came up with after I learned how to manipulate the headers of GeoTIFF's (GeoTIFF Tools helps) and I had found geo-calibrated photomaps on the map server of the Davenport Iowa area of the Mississippi River before they built the locks and dams. All the wingdams back then were emergent type meaning they came out of the water in the summer but after they raised the pools to make a 9 ft channel they are nearly all below the water all the time. So I overlayed a current photomap with transparency over the 1930's photomaps and you can clearly see the wingdams and other structure. The LIDAR maps have only been available a couple of years for this part of the state so that was the first chance to try overlaying to a LIDAR and I really liked the results. DEM maps are just too coarse and I map at 1 pixel/meter.

Open them up in a good photo viewer and make sure they are full size, those are 4640 pixels x 2925 pixel maps. For the final maps I'll redo everything using the photomaps from this year (And probably work a 5000 x 3000 meter map) that should show the lake at full level. I'm probably going to drive out there later today and see if they got the main ramp back open and take some pictures, probably the last before the lake tops off sometime this summer. The photos of the lake I've taken the last couple of years are not just to document it filling but also so I can identify down the road what I'm seeing on my fish finder and to place man made structure in the proper locations.

Lost Grove Lake Mapping Project

I just wanted to say after 5+ weeks of testing and usage this works better than any setup I have used, even better than a Combo chartplotter/fishfinder although that probably has more to do with the hardware setup (Remote antenna(s) almost directly over the tranducer(s) so no need to calibrate for distance or any delays using separate USB ports for each function)

The Lake finally topped off a few days ago and now the serious work begins

This is a test run from a couple of weeks ago and the data (tracks) are not spaced closely enough but it is still pretty good and gives an idea of what I’m going for here

The Lake finally topped off a few days ago and now the serious work begins

Fishing? 8)

Yup a brand new 400 acre lake specifically designed for fishing and also designed to keep silt and fertilizer runoff out of it (A huge problem with manmade lakes in Iowa or other states on a prairie)

Several varieties of sunfish, crappie, walleye, tiger muskies, bass and channel catfish .... I acquired some old concrete sewer pipes of various sizes and had them laid out for large spawning catfish (catfish hotels) and someone must have liked the idea because they put down even more in another place although some are too big of a diameter for spawning but I'm sure they'll make good cover for other species

If you look on the blog I started for the project you can see some of the basic background maps I made for plotting depths over

The Lost Grove Lake Mapping Project

Pictures of the Arduino based GPS and NMEA multiplexer with a 3D printed enclosure and mount can be found on my main blog

The Netbook2Chartplotter Project

No one has as nice of maps for this lake as I have already and I haven't even started marking all the manmade and natural structure or the depth markings. There is some other guy that's mapping the lake using a Lowrance HDS plotter/finder and putting it on the Lowrance Insight Genesis website but he doesn't look to know what he is doing plus with Insight all you get I guess is a Google map which is 2 years old and the lake wasn't even half full yet. It doesn't look like he added any shoreline data either which alone would help the accuracy of his map. I showed him the version of mine that runs on Android devices and he wanted to know how I was able to do that and I tried to explain but I could see he was lost after the first sentence. It takes 4 programs to build the maps not counting the depth mapping software ... QGIS, Geotiff Tools, GIMP, and OK Maps (All Open Source or free) OK Map is very impressive considering the cost (free) and it does most the things OziExplorer does ($109) and a few things it can't. Because of the cost and really good documentation I highly recommend it for someone who wants to get their feet wet making their own GPS navigation Maps (Plus maps downloaded from Google, Bing, Mapquest and Open Street Maps online and saved in the program) without being out anything but some time. It's also very good for making tiled maps for Orux maps (Android) and Garmin custom maps. I liked it so much I sent him a $25 donation and I don't do that very often because I'm kinda cheap (and why I make a lot of my own gear)

Nice work! Thanks for the mapping software suggestions. There’s nothing like knowing the combination of tools that work together nicely.

It looks like you found something, though:

Titanic.jpg

Cheers,
/dev

I put a slightly modified version in a new mapper 'bot I'm building because of all the flooded trees and brush making it nearly impossible to get a boat into some areas much less cover it well enough to map it. They planted brush and small trees back in the early '00s with an expected fill date of summer 2009 but Feds wanted a bigger dam so the fill date was pushed to 2015 and since the trees had 6 years longer to grow it's a bit of a mess although environmentally it's better for the fish (Fish growth rates out there are off the charts for this far north)

I call it the W.A.S.P - the Wireless Aquatic Science Platform and the framework is based on a project out of Spain (Thank heavens for Chrome's built-in translator) called the JALC Boat. I have some preliminary stuff up on my blog. It even has a Bluetooth link on it so I can get some live real time feedback

http://netbook2chartplotter.blogspot.com/

I also should note that logging 3 NMEA sentences seems to be about the limit, when I added a 4th sentence (water temp) I only got the finder data every 2 seconds instead of every 1 second in step with the GPS. However the output from the USB (or Bluetooth) of 6 sentences (3 from the finder, 3 from the GPS) seems to be unaffected so I am assuming that's because the 'quiet time' isn't long enough for the slow writing SD card if you try to push too much data to it. I didn't really have time to explore it more since my goal was to get the 'bot up and running as fast as possible so I could do some tests before it got too cold to justify putting a bunch of time in it this winter. I didn't even start the project until mid-August and initial tests (without transducer) didn't take place until mid-September but I did get enough time in with it to satisfy my Proof of Concept requirements to justify fully developing it over the winter. It's mostly a redesign and reprinting of the plastic parts which is the cheapest part so that leaves me time to develop some sort of APM unit for it so it will automagically map by self navigating a series of waypoints. I also want to develop a version with water quality and other environmental sensors.