Subroutine causing loss of serial data coming from a GPS. {SOLVED}

This problem has been driving me nuts for most of the day, because i can't find any logical reason for it to do what it does.

summary of the problem: I have a project that is acting as a sensor node with gps location, LCD GUI, SD logging, and xbee networking. I had everything working except the LCD in the last program revision. Upon adding the LCD GUI programming in the form of a subroutine LCDmain() to the main loop(), I start losing data from the GPS. Namely the last 5-10 characters of the GPRMC sentence. When I comment out the LCDmain() subroutine the program works again, even if i also run it once in the setup() section. Im running the baud rate at 57600 right now, but it does the same thing at 9600.

The incorrect string:

$GPRMC,232005.000,A,42xx.2558,N,071xx.1477,W,0.00,208.06,240$GPGGA,232006.000,42xx.2558,N,071xx.1477,W,1,10,0.87,57.4,M,-33.2,M,,*65
$GPRMC,232006.000,A,42xx.2558,N,071xx.1477,W,0.02,208.06,240

The correct string:

$GPGGA,232803.000,42xx.2596,N,071xx.1464,W,1,10,0.87,61.1,M,-33.2,M,,68
$GPRMC,232803.000,A,42xx.2596,N,071xx.1464,W,0.06,208.06,240614,,,A
72

The GPS Sentence Subroutine:

//=======================================================================================================
//Get the GPGGA or GPRMC sent
void Get_Sent()
{
  
    if (Serial1.available()>0)                                  //Is there Data waiting from the GPS?
  {
    CharIn = Serial1.read();                                    //Retrieve the character
    Serial.print(CharIn);                                       //Print the character to the serial monitor **for troubleshooting purposes**
    if ((CharIn == '

The Main Loop():

//=======================================================================================================
//Main loop
void loop()
{
  while ((Serial.available()>0)||(Serial1.available()>0)||(Serial2.available()>0)||inProc)
  {
    Serial_to_GPS();
    Get_Sent();
    Xbee_cmd_get();
  }
//  Serial.print("yes");
  Extract_Sent();
  print_time();
  send_data(Cmd_Check());
  LCDmain();
  Ready_Data();
//  loop_timer();
  if ((minute()%2 ==0) && (second() == 0))
  {
    En = true;
  }
}

The LCDmain() code is too large at the moment to post, has very few comments, and, at this point, has no interaction with any of the other parts of the program.) || (InSent != NMEAsentLayout))          //If there is a new sentence incoming, or a sentence already in transmission ($ designates the start of the sentence)
   {
     if (CharIn == '
[/code]

The Main Loop():

§_DISCOURSE_HOISTED_CODE_1_§

The LCDmain() code is too large at the moment to post, has very few comments, and, at this point, has no interaction with any of the other parts of the program.)                                        //If this is the beginning of a new sentence
     {
       inProc = true;                                          //inProc locks the serial coms while loop
//        Serial.println("inProc = true");                        //for Troubleshooting
       InSent = "";                                            //Clear the incoming sentence string
//        Serial.print("`");                                      //for Troubleshooting
     }else if (CharIn == '')                                  //If the incoming character is a '' indicating the end of the sentence
     {
       if (InSent.substring(0,5) == SentPrefA)                 //if the first five letters of the sentence are "GPGGA"
       {
         GgaSent = InSent;                                     //move the finished sentence into the GPGGA sentence String for later parsing
//          Serial.println(GgaSent);                              //for Troubleshooting
//          Serial.println("~");                                  //for Troubleshooting
       }else if (InSent.substring(0,5) == SentPrefB)           //if the first five letters of the sentence are "GPRMC"
       {
           RmcSent = InSent;                                   //move the finished sentence into the GPRMC sentence String for later parsing
//          Serial.println(RmcSent);                              //for Troubleshooting
//          Serial.println("~");                                  //for Troubleshooting
       }
       InSent = NMEAsentLayout;                                //put dummy sentence into the incoming sentence string
       inProc = false;                                         //unlock the serial coms while loop
//        Serial.print("~");                                      //for Troubleshooting
//        Serial.println("inProc = false");                       //for Troubleshooting
     }else if (InSent.length()<= 100)                          //If the sentence has not gotten too long(indicating a problem)
     {
       InSent.concat(CharIn);                                  //concatonate the character to the incoming sentence string
//        Serial.print(CharIn);
     }else
     {
     InSent = NMEAsentLayout;                                  //put dummy sentence into the incoming sentence string
     inProc = false;                                           //unlock the serial coms while loop
//      Serial.println("inProc = false");                         //for Troubleshooting
     }
   }else                                                         //if ther is data incoming, and it is not the start of a new sentence and there is no sentence currently in transmission
   {
//      Serial.println(" Junk");                                  //for Troubleshooting
//      Serial.print(CharIn);                                     //for Troubleshooting
   }  
 }
 
}
[/code]

The Main Loop():

§_DISCOURSE_HOISTED_CODE_1_§

The LCDmain() code is too large at the moment to post, has very few comments, and, at this point, has no interaction with any of the other parts of the program.

Without all the code I can't say anything. But if you say that the problem is theLCDmain() function and you don't send it, I don't know how do you think I can see anything wrong in that function.

I kinda figured that'd be a problem, but as I said, the LCDmain has absolutely no interaction with the rest of the program except to take up space and cycle time. I'll try to post a copy of the code later once I get some of the personal details changed and some comments made.

I had this problem once, and it happened because the gps device itself was sending its data too fast and overwriting its own data.
Your code seems to be very convoluted and obscure. If you just want a textual copy of the data, there are much more simple ways to achieve that. And if you want to extract the actual data content from the data, ditch all that string stuff and use tinygps.

The LCDmain() code is too large at the moment to post, has very few comments, and, at this point, has no interaction with any of the other parts of the program.

But it presumably prints things including text. Could that be causing a problem with memory shortage when it is included ?

Could that be causing a problem with memory shortage when it is included ?

That would be my guess, since the snippet uses Strings (which piss away memory) and does not use the F() macro to keep string literals out of SRAM.

Your incoming string is being truncated at 64 characters, which is the size of the serial input buffer. It's probably happening because you're going too long without reading from the serial port. Try to speed up the rest of your code so you can get back to reading the serial port faster, or slow down the serial speed so that the transmission takes longer to arrive so that you can get back to read it before the buffer has overflowed.

I'm looking into the string memory usage problem, and I may have to go with char arrays, or include string.h. Stability is paramount for this project.

I'm not familiar with F(), what its used for and how to use it. As you can surely tell, i'm not a super C programmer. I had a basic course for Electrical engineers in college, but thats it, and the rest I learned on my own. I am going have a look at the reference page for F(), but advice here is helpful.

another detail, if it wasn't noted already, is that I'm using an arduino mega2560.

it has been several months since I looked at tinyGPS or tinyGPS++, so all I remember at this point is that I had a reason for not using it. I'll look into it again just to be sure. Used to be an NMEA.h library out there that I used in some older projects. That was awesome, but it seems that it is no longer compatible with the newer IDE.

Your lcd routine is blocking?

That would explain things.

If you re-write it as non-blocking then it shouldn't hold up the serial
handling from the GPS unit.

Aim for anything called from loop() to return within a few ms if you can,
so that everything gets timely handling.

However if memory exhaustion is the problem the symptoms can be many and
varied and it can't be ruled out.

I just put a timer on the portion of the code outside of the serial coms and it is taking about 104600 microseconds to run with the lcd code. without the LCD code it runs at 2800 microseconds... doing a little math... at 57600 baud i can have code run up to a little over 8.5ms outside of the serial coms. 64*8/57600 = .00888888s

I also looked at how full the serial rx buffer (64bytes) was at the beginning of the sentence, and it was full, so data was being lost through a buffer overrun... That was being caused because the code couldn't get back to the serial coms in time. Can I make the buffer larger? now that i know the problem, and can fix it, this really isnt a big deal, but just for kicks...

some of you guys were spot on. I thought I had ruled that out as a possibility, until one of your comments pushed me out of my paradigm to see it from a different direction. Thank you!

I was going to ask about serialEvent(), but upon further reading, this is not truly interrupt driven, and so would have the same problem...

It would be possible to modify the hardware serial library and make the buffer bigger, but it's a horrible way to address the problem. Better to design your code so that it can read from the serial port more frequently, and if necessary slow the serial port down to give your sketch more time to get round to reading it. What is happening in the LCD code that is taking so long?

I started reviewing the lcd code, and found a delay(100) hidden there that was causing the problem (I wasn't familiar with the code. One of my coworkers wrote it.). I commented it out, and made a note that the code needs reworked to function without delay()s.