SDCard/Software Serial library incompatibilities?

Using an Arduino Pro Mini 16MHz I'm trying to read logged data from an Adafruit GPS Breakout and write it to an SDCard. The data comes out of the GPS at 9600 baud (no flow control is available) in records around 230 bytes long which I'm reading with SoftwareSerial. The Adafruit GPS library (which I've had to modify) double buffers these using two 120 byte buffers. As soon as a buffer is full I write it to the SDCard.

A test sketch shows that the SDCard can soak up buffers of that size in a few mS each with the occasional one taking up to 5mS. There should therefore be no problems with the SDCard being able to keep up. Nevertheless, I'm loosing a lot of buffers.

I assume both SoftwareSerial and the SDCard library (or rather the underlying SPI library) will need to disable interrupts for part of the time in order to do real-time or time critical stuff and so I wonder if they perhaps don't sit well together.

Can anyone with a deeper understanding or greater experience offer any insights?

Regards - Philip

Are you simply trying to echo the NMEA strings to the SD card? Or are you processing the strings and writing the data in a different format? Is the 230 bytes of GPS data comprised of three NMEA strings? How often is it received, once per second or more frequently?

The Adafruit GPS module has a self-logging function in which it logs data to internal flash memory. You can then dump it out, and it comes out as lines consisting of a few protocol characters and 24 comma-separated hex coded 32 bit values, making up the line length of 232 characters.

I'm trying to catch this data, coming at 9600 baud using SoftwareSerial and double-buffered in two 120 byte buffers (no flow control available), and write it to the SDCard. I appear to be loosing a lot of buffers of data. Having included the SDCard, SPI and Adafruit GPS libraries, I had to optimise strings with the F() macro to get it to eliminate the IDE warnings about stability and get it to work at all, so little chance of adding any more bufferage. For the moment, I'm stuck.

Regards - Philip

I see. I know that particular GPS has that feature but I haven't tried it (yet).

So when downloading the locus data the GPS transmits 230 byte strings without any gaps between the strings? If that's the case I can understand why you are having a problem using SoftwareSerial. When receiving it hogs about 95% of the processor bandwidth. There are alternatives to SoftwareSerial that might solve your problem.

I don't understand why you are using 120 byte buffers if the strings are 230 bytes long. Is that simply because that's what Adafruit uses in their logging example?

Yes, each line is like this:

$PMTKLOX,1,84,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF,FFFFFFFF*64

except that it has hex data instead of FFFFFFFF. The Adafruit GPS library uses 120 byte buffers and I'd probably run out of RAM if I doubled them.

If SoftwareSerial is hogging the CPU then I can well imagine the SDCard library won't get a look in. I thought that's why the old SoftwareSerial was bad, and the NewSoftwareSerial (which I presume I'm using, as the default) worked on interrupts.

I think there's an AltSoftwareSerial which I might try, but it wasn't very clear (to me at least) what the differences in operation were. I'm trying to modify it to use HW Serial for the GPS and SoftwareSerial for commands and responses to and from my sketch, but something's broken for the moment and it's not even working as before, with the #define supposedly configuring out the changes.

Regards - Philip

I thought that's why the old SoftwareSerial was bad, and the NewSoftwareSerial (which I presume I'm using, as the default) worked on interrupts.

The "NewSoftwareSerial" which is the current library in the IDE does use interrupts but as soon as it sees a start bit, it disables interrupts and clocks in the bits using an AVR delay loop. It helps but no CPU time is available if you get a steady stream of characters.

AltSoftSerial would make a big difference. It uses about 5% of the bandwidth versus 95% for SoftwareSerial. But you’d have to cut a trace and do some soldering since it requires specific pins (pin 8 for RX and pin 9 for TX on an Uno).

Alternatively, you could use the software serial code I wrote specifically for a GPS. You can use any of the pins. It consumes about 7% of the processor bandwidth at 9600 baud.

I tried the Adafruit LOCUS sketches. After starting the LOCUS and letting it acquire data for a while I ran their dump sketch. It didn’t work very well, dropping a lot of characters. So I modified it to use my soft serial and got it to output the LOCUS data. Then I added SD so it would generate a file. It seems to work. Maybe it will help you.

// LOCUS dump code for the Ultimate GPS Logging shield
// This writes the LOCUS data to a file "LOCUS_NN.TXT", where the NN are digits 00-99.
// It uses the gSoftSerial library.
//
#include <gSoftSerial.h>
#include <SD.h>

#define SD_CARD_CS      10
#define PMTK_SET_NMEA_OUTPUT_OFF "$PMTK314,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0*28"

gSoftSerial gps(8, 7);
File logfile;
unsigned long ms;    // used to guess when there's no more output (easier than parsing)

void setup()  
{    
  Serial.begin(115200);
  Serial.println("\nLOCUS dump sketch\n");
  Serial.print("Type any key to start...");
  while (!Serial.available()) {}              // wait for a key
  Serial.println();
  
  if (!SD.begin(SD_CARD_CS)) {
    Serial.println("Couldn't init SD card");
    while (1) {}
  }
  
  char filename[15];
  strcpy(filename, "LOCUS_00.TXT");
  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = '0' + i/10;
    filename[7] = '0' + i%10;
    if (!SD.exists(filename)) {
      break;
    }
  }

  logfile = SD.open(filename, FILE_WRITE);
  if(!logfile) {
    Serial.print("Couldnt create "); 
    Serial.println(filename);
    while (1) {}
  }
  Serial.print("Writing to "); 
  Serial.println(filename);

  gps.begin(9600);
  gps.println(PMTK_SET_NMEA_OUTPUT_OFF);

  while (gps.available())
     gps.read();                // flush gps output

  delay(1000);
  gps.println("$PMTK622,1*29");    // request LOCUS dump
  Serial.println("\nDUMP DATA:\n");
  
  ms = millis();                // current time
}

void loop()
{  
  if (gps.available()) {
    char c = gps.read();
    UDR0 = c;  
    logfile.write(c);
    ms = millis();
  } else {
    if ((millis() - ms) > 500) {    // if it has been a while since any data arrived it's probably done
      logfile.close();
      Serial.println("\nAll done.");
      while (1) {}
    }
  }
}

gSoftSerial_2Jul2015.zip (7.26 KB)

Using HWSerial for the GPS and SoftwareSerial for chatting to my sketch works, and I can dump LOCUS data to the SDCard reliably. But I've had to disable reflection of raw NMEA packets as it works by simply dumping bytes on the UART transmit register, which you obviously can't do with SoftwareSerial. Writing them a byte at a time using write() might work given that they're coming in on the UART which will certainly buffer one character at 9600Baud, which ought to be sufficient to let SoftwareSerial write one character at 115kBaud. But just for turning logging on and off, dumping log data to the SDCard and clearing logged data, it's not really needed. The more serious inconvenience is having to swap serial connections between programming and running.

Thanks jboyton for gSoftSerial. It took a bit of work on the Adafruit GPS library in order to use it, but it still won't compile, giving inscrutable errors like multiple definition of `__vector_3' (also vector_4 and _5). A bit of googling indicates that it's a clash between libraries - possibly due to reuse of timer2? I'm running on a Pro Mini at 16MHz.

Regards - Philip

The Adafruit GPS library is tightly bound to SoftwareSerial. The errors you got are due to the fact that both gSoftSerial and SoftwareSerial declare the pin change interrupt vectors.

The Adafuit GPS library has a number of limitations. I highly recommend using the NeoGPS library instead. Or if not that then TinyGPS. These libraries don't care which serial you use. And they parse on the fly so you don't need those big buffers that the Adafruit library requires.

Cheers.

As I said, I modified the Adafruit GPS library to use gSoftSerial, but I used #ifdefs in both that and my sketch to configure which softserial library to use so I could try before I buy. The problem seemed to be due to the way the IDE collects bits together to stuff into gcc, which seems to result in #includes being included despite being excluded by #ifdefs, and a resultant clash with both softserial libraries being included. Commenting out the ones I didn't need gave a clean compile but I haven't tried it as I've moved on to the next stage in my project, using HWSerial for the GPS instead as a work-around.

Regards - Philip

For the record, the original problem of lost records seems to have been a bug in the double buffering, which I'd had to modify on account of the Adafruit GPS library's quirky habit of delivering records with newline at the beginning instead of the end, and the way they delivered records and reset the recvdflag. My code now seems to be working reliably with the standard SoftSerial and SC libraries.

Regards - Philip

Glad you got it to work.

Would you mind sharing the sketch? Adafruit might appreciate it as well.