NMEA multiplexer and logger

I am building a device to multiplex two 4800 baud NMEA data streams, echo them to an output port and log them to an SD card. It is based on a Mega 2560 and the Adafruit Datalogger. The NMEA is interfaced through a MAX232 which connects to Serial1 and Serial2 on the Mega. The I2C connections to the Adafruit datalogger have been made by jumpers to the SDA and SCL pins on the Mega.

I have tested the device with a single data stream, and it works fine with the data from a Garmin Etrex connected either to port 1 or to port 2. With the data from an Advansea T50, I get some corrupted sentences. As these always seem to happen after 62 or so characters I infer these are buffer overflows. When I connect two data streams, I get more corrupted sentences, presumably associated with more buffer overflows - I think this may be tied up with the size of the serial buffers, and the time taken to write to the SD card.

I am using Arduino 1.02 IDE.

Is there any way to increase the size of the serial buffers, or to clear the data quicker from them?

The code I started off with read a complete sentence from a port once it had data available. This approach probably blocks the process until that sentence has been received. I have also tried to re-configure the code to read individual characters from each port as they become available, but I am having problems with this also.

Has anyone done anything comparable and found a fix?

Would an Arduino DUE be any better for this application?

Extracts from the "complete sentence reading" code are shown below.

//-------------------------------------------------------------------------- // Main loop segment // Alternately checks ports 1 & 2, if data present reads and logs sentences. void loop() {

if(Serial1.available()) { readline1(); logline(); }

if(Serial2.available()) { readline2(); logline(); } } //----------------------------------------------------------------------- void readline2(void) {

buffidx = 0; // start at begninning while(1) { c=Serial2.read(); if (c == -1) continue; if (c == '\n') continue; if ((buffidx == BUFFSIZ-1) || (c == '\r')) { buffer[buffidx] = 0; return; } buffer[buffidx++]= c; }

}

//--------------------------------------------------------------------------

The readline1 routine is similar to the readline2 routine above. The logline routine checks the checksums and filters out unwanted sentences, and sends the wanted ones to a fileline routine for saving to the SD card. The buffer character arrays have a max size of 90.

That code will block one or other of the input streams as you say.

But I'm surprise that you get anything sensible because your loop sis essentially

while(1) {
      c=Serial2.read();
  ...
      buffer[buffidx++]= c;
  }

Except for the first byte that causes you to enter the function you are not waiting for characters to be available.

Something like this would be more appropriate I think.

void loop()
{

  while (Serial1.available()) {
    x = read_and_buffer_char(1);
    if (x = GOT_LINE) 
        logline();
    }

  while (Serial2.available()) {
    x = read_and_buffer_char(2);
    if (x = GOT_LINE) 
        logline();
    }
}

Rob

Thanks for the reply.

The code in the readline loops came from a sketch I found, I assume the c==-1 deals with no new character available, so it just loops until the sentence is complete - when I tried it it worked!

I have rewritten the main segment along the lines I think you are suggesting:-

// Main loop segment
// Alternately checks ports 1 & 2, if data present reads and logs sentences.
void loop() 
{ 
   
// read1:
  while(Serial1.available()) {
      c1=Serial1.read();
      if (c1 == -1) continue;
      if (c1 == '\n') continue;
      if ((buff1idx == BUFFSIZ-1) || (c1 == '\r')) {
        buff1[buff1idx] = 0; 
        strcpy(buffer, buff1); Serial.print(buff1idx); Serial.print("   ");Serial.println(buffer);
        buffidx = buff1idx;  logline(); 
        buff1idx=0; continue;}
      buff1[buff1idx++]= c1;
    }

// read2:  
  while(Serial2.available()) {
      c2=Serial2.read();
      if (c2 == -1) continue;
      if (c2 == '\n') continue;
      if ((buff2idx == BUFFSIZ-1) || (c2 == '\r')) {
        buff2[buff2idx] = 0; 
        strcpy(buffer, buff2); Serial.println(buff2idx);  
        buffidx = buff2idx; logline(); 
        buff2idx=0; continue;}
      buff2[buff2idx++]= c2;    
    } 
      
}
//----------------------------------------------------------------------------

Assume your code is intended to represent the kind of reading and buffering above, but is not the actual code to make it work. I assume that GOT_LINE is also representative of detecting '/r' as above.

As the code should now only read when a character is available, I can probably drop the c==-1 test from the code.

With this new code I am still getting sentences corrupted by apparent buffer overflows that occur during writes to the SD card.

I have found a serial library: http://code.google.com/p/rtoslibs/downloads/detail?name=SerialPort20130222.zip&can=2&q=

This library suggests that it can assign different buffer sizes to the serial ports. I'm still trying to figure out what it does, but I hope I can try it shortly and see if it makes a difference.

Any other suggestions welcome.

I'm not familiar with the SD library but I believe there are large delays involved with writing to an SD card, this may be causing you grief.

You can increase the buffer size by changing the

define SERIAL_BUFFER_SIZE 64

line in HardwareSerial.cpp, but if the SD lib blocks interrupts you are no better off. If the lib doesn't block interrupts that may be all you need to do.

You should test one thing at a time though, does the code work if you don't write to the SD card?

Also at what speed are you printing? That may block the reception as well.


Rob

Thanks for the reply.

I still got the symptoms of buffer overflow with the flushing to SD disabled.

I have only just found out how to locate HardwareSerial.cpp - I'm using Arduino 1.02 on OSX, so its all hidden inside the package until one opens it!

However - I have changed libraries to use SerialPort [so I can easily specify buffer sizes] and SdFat [as SD is not compatible with SerialPort]. Using 256 byte buffers I'm not getting the overflows I was seeing even with just one data stream.

With SdFat I have used different options to open the logfile, so writing is faster.

if(!logfile.open(filename, O_RDWR | O_CREAT | O_AT_END)){
        error("couldnt create file");}

I have also modified the code for writing to SD so that it writes only when there are nearly 512 bytes to write. After the initial write to SD, the subsequent ones now seem to take only 40-60 ms, instead of 160-300 ms previously.

//----------------------------------------------------------------------------
// Write & echo a selected line 
void fileline(void){
               buffidx ++;
               buffer[buffidx] = '\n';                        // add newline
               buffidx ++;
               buffer[buffidx] = '\r';                        // add carriage return
               Sdbuf = Sdbuf + buffidx;                       // size of SD buffer if this sentence is written
               if(Sdbuf>500) {                                // If buffer nearing full, sync to SD card
                 digitalWrite(led,HIGH);
                 NewPort.print(millis());NewPort.print("   ");
                 logfile.sync(); Sdbuf = buffidx;             // Reset size of SD buffer
                 NewPort.println(millis());
                 digitalWrite(led,LOW);
                 }
               logfile.write((uint8_t *) buffer, buffidx);    // write the string to the SD buffer
               NewPort1.println(buffer);                      // echo to port1
               }

//--------------------------------------------------------------------------

Upping the serial monitor speed to 115200 has helped reduce the time committed for displaying debugging information.

I'll be able to test it with the intended two input streams shortly, but the indications are good.

Thanks for your suggestions.

I have now tested this with both the input streams running and it works just fine. The SerialPort and SdFat libraries seem to be just what was needed.

Good to see you got it working then.


Rob

Could you post the entire code please.

The full code is too big to be posted on the forum. As you are interested in the bit that reads from the input ports, I have attached that [it is the ‘void loop()’ segment].

If you want the rest of the code, can you give me an email address to send it to?

Code attached:-

//-------------------------------------------------------------------------- 
// Main loop segment
// Alternately checks ports 1 & 2, if data present reads and logs sentences.
void loop() 
{ 
  
    if ((millis() - mv) >= VOLT_INTERVAL) {mv = millis(); logvolt();}
  
// read1:
  while(NewPort1.available()) {
      c1=NewPort1.read();
      if (c1 == -1) continue;
      if (c1 == '\n') continue;
      if ((buff1idx == BUFFSIZ-1) || (c1 == '\r')) {
        buff1[buff1idx] = 0; 
        strcpy(buffer, buff1); 
        // NewPort.print(buff1idx); NewPort.print("   ");NewPort.println(buffer);
        buffidx = buff1idx;  logline(); 
        buff1idx=0; continue;}
      buff1[buff1idx++]= c1;
    }

// read2:  
  while(NewPort2.available()) {
      c2=NewPort2.read();
      if (c2 == -1) continue;
      if (c2 == '\r') continue;
      if ((buff2idx == BUFFSIZ-1) || (c2 == '\n')) {
        buff2[buff2idx] = 0; 
        strcpy(buffer, buff2); 
        // NewPort.print(buff1idx); NewPort.print("   ");NewPort.println(buffer);  
        buffidx = buff2idx; logline(); 
        buff2idx=0; continue;}
      buff2[buff2idx++]= c2;    
    } 
    


  if(bmplog==1) {
  if ((millis() - mm) >= MET_INTERVAL) {mm = millis(); logmet();}}
  
  // digitalWrite(led,HIGH);delay(50);digitalWrite(led,LOW);delay(50);
  
}
//----------------------------------------------------------------------------