[SOLVED] Problem data logging with TinyGPS++ and SdFat

A newbie here and my first ever post here. Please be supportive. And please bear with any silly mistakes I may have made in this post. So first thing first, here is my objective: I want to log GPS location and ancillary data at relatively low sample rate (say once every 3-5 seconds) on a MicroSD card.

And here is my hardware:

  1. A strip board based ATmega328p running at 8Mhz internal clock, powered by regulated 3.3V (otherwise running perfectly with most other sketches),

  2. GPS breakout (I alternate between a uBlox Neo-6 from eBay, and Adafruit Ultimate GPS, basically no significant difference here, both perfectly spitting valid NMEA sentences and getting fix in short time)

  3. A microSd breakout (which simply exposes MiscroSD pins to a header friendly 0.1" spaced pins, absolutely no other circuit involved)

  4. An ordinary 2GB OEM MicroSd card formatted by SD formatter utility (otherwise perfectly working with another sketch at higher sampling/logging rates, i.e., combined DHT11 Humidity&Temp sensor + SR04 Ping sensor at 8-10 samples per second).

Now here is what my problem is:
SD card is detected, file opened, data logging starts - but cannot proceed as expected. Any of these things happen randomly- file content gets garbled (unexpected things including extended ASCII symbols get written, no consistent end-of-line, even some of my Serial.print stuff gets written here when I keep Serial output ON).

I assume, I have no hardware, electrical, electromagnetic, wiring, bad SD card, or wrong formatting issues based on consistent successes I have with other sketches. That brings me down to this particular GPS data logger sketch. It contains SoftwareSerial.h, TinyGPS++.h and SdFat.h. I know all three are well written, widely used, well tested and well run libraries, there shouldn’t be any question. So, my codes must have the root of the problem, unfortunately I cannot figure out what is the culprit.

I have checked available RAM at runtime and optimized RAM usage as much as I could. Now It shows FreeMem()=798 bytes in Setup(), and FreeMem()=792 bytes in Loop(). I have no clue what else I can do to solve my problem.

Here goes my code:

/*
 GPS data logger using TinyGPS++ and SoftwareSerial libraries and examples by Mikal Hart
 Sketch written by: Sayedur R Chowdhury, Dec 10, 2013

 Data to be logged: Date, Time, Lat, Lon, Altitude, DOP, Speed, Bearing

 Indicator LED:
  	Uses one common anode RGB LED. Connections:
   	Anode 	= pin 6
    	Red  	= pin 5
 	Green 	= pin 7
  	Blue 	= pin 8
  
 GPS connections:
  	Rx 	= pin 2
   	Tx 	= pin 3
 
 MicroSD connections:
   	CS 	= pin 10
    	MOSI 	= pin 11
     	MISO 	= pin 12
  	SCK 	= pin 13

 */
//#define SERIALOUT

//#include <MemoryFree.h>
#include <TinyGPS++.h>
#include <SoftwareSerial.h>
#include <SdFat.h>

//definitions fo SoftwareSerial GPS
static const byte Rx = 3, Tx = 2;  //rx,tx (3,2) on the Arduino side connects to Tx, Rx of GPS respectively
static const int baudrate = 9600;

//defintion for RGB LED
static const byte commonAnode=6, R=5, G=7, B=8;

//definitions for SD card and file
SdFat sd;
SdFile datafile;
static const int SD_CS=10;

bool writeHeader=1;

//TinyGPS NMEA stream 
TinyGPSPlus nmea;
//TinyGPSCustom pdop(nmea, "GPGSA", 15);  //takes up 50 bytes of SRAM

//serial connection to the GPS device
SoftwareSerial gps(Rx, Tx);


void setup()
{ 
	#ifdef SERIALOUT
	Serial.begin(9600);
	#endif

	//set up RGB led pins
	pinMode(commonAnode, OUTPUT);
	pinMode(R, OUTPUT);
	pinMode(G, OUTPUT);
	pinMode(B, OUTPUT);
	digitalWrite(commonAnode, HIGH);
	digitalWrite(R, HIGH);
	digitalWrite(G, HIGH);
	digitalWrite(B, HIGH);

	gps.begin(baudrate);
	pinMode(SD_CS, OUTPUT);

	if (sd.begin(SD_CS, SPI_HALF_SPEED)) {
		//indicate success by long flashing ok led twice
		flashLED(1000, G);
		smartDelay(500);
		flashLED(1000, G);
		#ifdef SERIALOUT
		Serial.println(F("SD Ok"));
		#endif
	}
	else {
		//indicate SD init error by long flashing error-led once
		flashLED(1000, R);
		#ifdef SERIALOUT
		Serial.println(F("SD Error"));
		#endif
	}  //SD.begin ends

	#ifdef SERIALOUT
	Serial.print("freeMemory()=");
	Serial.println(freeMemory());
	#endif
}    //setup ends

void loop()
{
	unsigned int pastCharProcessed=nmea.charsProcessed(), recentCharProcessed=0;
	int flashTimer=0;
	long previousMillis = 0;
	unsigned long currentMillis = millis();
	//SdFile datafile;

	//smartdelay at the beginning
	smartDelay(3000);
	//check new nmea.charProcessed because smartDelay has already elapsed some time
	recentCharProcessed=nmea.charsProcessed()-pastCharProcessed;

	flashTimer=currentMillis-previousMillis;

	//verify if GPS data coming in
	if (recentCharProcessed < 10) {    //not enough data coming in
		//flash errorLed every 3 seconds if no data is coming, diagnose wiring problem
		if (flashTimer > 3000) {
			previousMillis = currentMillis;
			flashLED(10, R);
			flashTimer=0; //reset timer
			#ifdef SERIALOUT
			Serial.println(F('No data'));
			#endif
		}
	}
	else { //expected amount of data is coming
		//verify if a fix is obtained
		if (!nmea.location.isValid()) {  //no fix yet
		//flash fixLed every 10 seconds if data is coming but no fix yet
			if (flashTimer > 10000) {
				previousMillis = currentMillis;
				flashLED(10, B);
				flashTimer=0; //reset timer
				#ifdef SERIALOUT
				Serial.println(F('Data Ok, no fix'));
				#endif
			}
		}
		else {  //fix obtained
			//GPS units generally have fix indicators, an indicator is not needed; comment out flashLed codes below 
			/*
			if (flashTimer > 30000) {
			previousMillis = currentMillis;
			flashLED(10, G);
			flashTimer=0; //reset timer
			}
			*/
			#ifdef SERIALOUT
			Serial.println(F('Fix obtained'));
			#endif

		//Debug: the following closing brace moved here from below to initiate loggin even before a fix
		//in the final sketch logging will start only when a fix is obtained
		} //!nmea.location.isValid()

			//now that a fix is obtained,  proceed onto logging data to SD card file

 			if (datafile.open("GPSDATA.CSV", O_WRITE | O_CREAT | O_APPEND)) { 
				//indicate file success by flashing OkLed once
				//following conditional LED flashing will change into unconditional green in final sketch
				if (nmea.location.isValid()) {
					flashLEDbicolor(20, R, B);
				} else {
  					flashLED(20, G);
  				}

				#ifdef SERIALOUT
				Serial.println(F("GPSDATA.CSV opened"));
				#endif

				//write CSV header only once at the beginning of the loop
				if (writeHeader==1) {
					datafile.println();
					datafile.print(F("GPS data logging starts at "));
					datafile.print(millis());
					datafile.println(F(" mSecs"));
					datafile.println(F("date,time,lat,lon,altitude,DOP,speed,course"));
					writeHeader=0;
				}

				//date-time
				if (nmea.date.isUpdated()) {
					datafile.print(nmea.date.year());
					datafile.print("/");
					datafile.print(nmea.date.month());
					datafile.print("/");
					datafile.print(nmea.date.day());
				}
				datafile.print(",");
				if (nmea.time.isUpdated()) {
					datafile.print(nmea.time.hour());
					datafile.print(":");
					datafile.print(nmea.time.minute());
					datafile.print(":");
					datafile.print(nmea.time.second());
					datafile.print(".");
					datafile.print(nmea.time.centisecond());
				}
				datafile.print(",");

				//location
				datafile.print(nmea.location.lat(),6);
				datafile.print(",");
				datafile.print(nmea.location.lng(),6);
				datafile.print(",");

				//altitude
				datafile.print(nmea.altitude.meters(),5);
				datafile.print(",");

				//DOP
				datafile.print(nmea.hdop.value(),5);
				datafile.print(",");

				//speed
				datafile.print(nmea.speed.kmph());
				datafile.print(",");

				//bearing
				datafile.print(nmea.course.deg());

				//last field saved, start a new line
				datafile.println();

				#ifdef SERIALOUT
				Serial.println(F("Data written."));
				#endif

				//close datafile
				datafile.close();
			} //if datafile succeeds
			else {  //datafile error

				#ifdef SERIALOUT
				Serial.println(F("Write error"));
				#endif

				//flash error LED
				flashLED(400, R);
			}  //datafile

		//Debug: the following closing brace moved upward to intiate logging even before a fix
		//} //!nmea.location.isValid() ends

	} //recentCharProcessed < 10

	#ifdef SERIALOUT
	Serial.print("freeMemory()=");
	Serial.println(freeMemory());
	#endif
}

// This custom version of delay() ensures that the nmea object is being "fed".
static void smartDelay(unsigned long ms)
{
	unsigned long start = millis();
	do 
	{
	while (gps.available())
		nmea.encode(gps.read());
	} while (millis() - start < ms);
}

//red LED indicator of Errors
static void flashLED(int duration, byte color)
{
	digitalWrite(color, LOW);
	smartDelay(duration);
	digitalWrite(color, HIGH);
}

static void flashLEDbicolor(int duration, byte color1, byte color2)
{
 	digitalWrite(color1, LOW);
 	digitalWrite(color2, LOW);
 	smartDelay(duration);
	digitalWrite(color1, HIGH);
	digitalWrite(color2, HIGH);
}

UPDATE

I have just had some success by turning OFF the Serial monitor output completely. Now data seems to be logging for some time. But as soon as the GPS gets position fix, it fails soon, not immediately but after writing a few records. Here is the latest output from my SD card:

GPS data logging starts at 5988 secs
date,time,lat,lon,altitude,DOP,speed,course
2000/0/0,0:0:0.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2000/0/0,14:23:46.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:49.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:52.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:55.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:58.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:1.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:4.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:7.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:10.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:13.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:16.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:19.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:22.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:25.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:28.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:31.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:35.0,0.000000,0.000000,4114,0.00000,0.00,0.00
2013/12/13,14:24:38.0,22.340637,91.829925,4114,0.00000,2.72,193.13
2013/12/13,14:24:41.0,22.340641,91.829933,4114,0.00000,0.63,190.74
2013/12/13,14:24:44.0,22.340642,91.829933,4114,0.00000,0.74,191.47
2013/12/13,14:24:47.0,22.340648,91.829933,4114,0.00000,1.26,191.35
2013/12/13,14:24:50.0,22.340652,91.829933,4114,0.00000,1.02,191.02
2013/12/13,14:24:53.0,22.340654,91.829933,4114,0.00000,0.91,191.56

Well, I may have forgotten to mention that I needed some guidance/expert advice/help/clues - whatever you want to call it; here I am in need of some help. Please!

Have you checked the SRAM according to this: http://playground.arduino.cc/Main/PROGMEM For me it sounds like a RAM issue...

Cheers, Christian

@Christian

Thanks for reply. I have checked RAM by the following code …

#include <MemoryFree.h>
...
...
#ifdef SERIALOUT
	Serial.print("freeMemory()=");
	Serial.println(freeMemory());
#endif

if you send the GPS output to your computer instead, can you see it ?

Are you getting gps readings, gps doesn't work very well indoors.

The bottom part of the sd file example you show, looks OK. You are in India. The first part is zeros, because it takes the GPS several seconds or minutes to start.

Actually, I should be paying more attention. You’re in Chittagong.

Do you know how GPS modules work ? It takes a second or two for it to detect the first satellite. That’s when it starts telling you the time and date. It then takes anything between 10 seconds and 10 minutes to download the satellite almanac and find at least 3 or 4 satellites. That is when you get a position fix. There is a code for this in the NMEA output.

If you don’t want those zeros in your file, don’t start storing data in your file until you get that fix.

The other problem I can see, is that 3 second delay at the beginning of loop(). If your GPS is sending you a string every second, that 3 second delay is going to cause a traffic jam of some kind. Try making that delay a lot smaller.

I am not too keen on all those datafile.print() function calls. I'd use sprintf() to write the result into a char array and then write the whole line to the datafile at once.

The other thing you might want to look into, is why does you altitude and hdop figure look like nonsense. Is there a type conflict there ?

michinyon: Actually, I should be paying more attention. You're in Chittagong.

Do you know how GPS modules work ? It takes a second or two for it to detect the first satellite. That's when it starts telling you the time and date. It then takes anything between 10 seconds and 10 minutes to download the satellite almanac and find at least 3 or 4 satellites. That is when you get a position fix. There is a code for this in the NMEA output.

If you don't want those zeros in your file, don't start storing data in your file until you get that fix.

Thanks michinyon. Yes, I am in Chittagong (Bangladesh), not in India :D. I think I know how a GPS works, because I have a relevant Masters degree and several PhD courses on GIS, Remote Sensing, Survey and GPS technologies. I am a professional in this field since 2002, and operated a dozens of kinds of GPS hardware from simple handheld to DGPS/RTK, to navigational, to USB, bluetooth... you name it. I have mentioned and is apparent in my code that for debugging purpose I have initiated data logging before the GPS hardware gets a fix.

Thanks anyways, but I am still in dark as to why the MCU gets crazy after some time. RAM is a suspect, I am expecting an an expert to tell me whether 792 bytes of free RAM should be considered adequate, marginal or inadequate when 512 bytes of SD blocks has to be written and when these three libraries are in action.

michinyon: The other thing you might want to look into, is why does you altitude and hdop figure look like nonsense. Is there a type conflict there ?

No it's because I had some mistake in my code (particularly the order of the fields in header row vs. the order of the fields in actual data rows), which I have now fixed.

With a little bit of minor tweaking here and there, and getting the device to start logging only after fix I have this output....

GPS data logging starts at 6516 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:19.0,22.340890,91.830078,0.00000,0,0.17,0.00
2013/12/13,17:13:22.0,22.340890,91.830078,0.00000,0,0.02,0.00

GPS data logging starts at 5826 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:31.0,22.340883,91.830070,0.00000,0,0.13,0.00
2013/12/13,17:13:34.0,22.340881,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:13:37.0,22.340879,91.830070,0.00000,0,0.02,0.00
2013/12/13,17:13:39.0,22.340879,91.830070,0.00000,0,0.02,0.00
2013/12/13,17:13:42.0,22.340877,91.830070,0.00000,0,0.13,0.00

GPS data logging starts at 5711 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:48.0,22.340875,91.830062,0.00000,0,0.09,0.00
2013/12/13,17:13:51.0,22.340873,91.830062,0.00000,0,0.06,0.00
2013/12/13,17:13:54.0,22.340873,91.830062,0.00000,0,0.13,0.00
2013/12/13,17:13:57.0,22.340873,91.830062,0.00000,0,0.04,0.00
2013/12/13,17:14:0.0,22.340873,91.830062,0.00000,0,0.04,0.00

GPS data logging starts at 5920 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:15.0,22.340877,91.830055,0.00000,0,0.07,0.00
2013/12/13,17:14:18.0,22.340877,91.830055,0.00000,0,0.11,0.00

GPS data logging starts at 5818 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:27.0,22.340877,91.830055,0.00000,0,0.07,0.00

GPS data logging starts at 5863 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:36.0,22.340879,91.830062,0.00000,0,0.20,0.00

GPS data logging starts at 5908 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:51.0,22.340885,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:14:54.0,22.340887,91.830070,0.00000,0,0.11,0.00
2013/12/13,17:14:57.0,22.340887,91.830070,0.00000,0,0.04,0.00
2013/12/13,17:15:0.0,22.340888,91.830070,0.00000,0,0.00,0.00
2013/12/13,17:15:3.0,22.340888,91.830062,0.00000,0,0.13,0.00

GPS data logging starts at 5918 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:15:19.0,22.340883,91.830070,0.00000,0,0.07,0.00

GPS data logging starts at 5898 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:15:25.0,22.340879,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:15:28.0,22.340877,91.830070,0.00000,0,0.06,0.00
2013/12/13,17:15:31.0,22.340877,91.830078,0.00000,0,0.07,0.00

than means the MCU is reseting randomly after every few records... please someone give me some guidance which direction I should take now.

I am not too keen on all those datafile.print() function calls. I'd use sprintf() to write the result into a char array and then write the whole line to the datafile at once.

SD data is buffered. So, writing one piece of information at a time to the SD buffer is better than writing to another buffer and then having that buffer added to the SD buffer.

    long previousMillis = 0;
    unsigned long currentMillis = millis();

Are you planning to support the concept of time going backwards? If not., all time variables are unsigned.

            Serial.println(F('Fix obtained'));

Which ONE key did you press to get that ONE character between the single quotes?

Have you fixed the issue with the overly large value in the call to smartDelay()?

Though I am not getting advice to the level I'd expected, I continue to share my experience in a hope that someone like me may be benefited.

I just got rid of the SoftwareSerial Library in my code and that saves additional 112 bytes of precious RAM, may be even more. This time I am saving FreeMemory() return value to SD card, because the hardware serial is occupied by the GPS. I don't know if I could save few more bytes if I stop writing FreeMemory() return value to SD card.

I wish I could get rid of TinyGPS++ and SdFat libraries as well, but those are at the heart of the operation. Does anyone know thinner alternatives to these libraries, or did someone benefit from subsetting the libraries?

Thanks a lot Paul.

I am not too keen on all those datafile.print() function calls. I'd use sprintf() to write the result into a char array and then write the whole line to the datafile at once.

I am actually not a C person, I tried to concatenate all the fields in one string, but those pointer-char-string related compiler errors compelled me to back out. Any help (code snippet) would be highly appreciated. But then, do you think that could improve my situation? Why and how? (Knowing the theory is always good :) )

Are you planning to support the concept of time going backwards? If not., all time variables are unsigned.

No I'm not. I shall modify if that helps. :)

Which ONE key did you press to get that ONE character between the single quotes?

There were some single quotes, but now I have changed all to double-quotes. Does this matter?

Have you fixed the issue with the overly large value in the call to smartDelay()?

I thought if I use long delay in smartDelay(), old values would be overwritten by new values that are constantly coming from GPS. Doesn't TinyGPS++ take care of that?

I'm sorry if my first advice seemed odd. I was looking at the first section of output you printed, and it looked like all zeros, I didn't see there was some actual data down at the bottom of the file, it was out of the viewing window.

You did not say you were a GPS expert.

If your program runs and then hangs up, or restarts, then it is likely to be one of three things. Program corruption ( unlikely on the arduino ) , running out of memory ( much more likely ), or some kind of power supply problem. How reliable is your arduino power supply ? Is there enough power to keep your GPS chip running ?

This TinyGPSPlus is new, I didn't see it before today, I just downloaded it and was playing with it myself. Are you using the very latest version of it ? Are you sure there is no memory leak in it? Try running it for a long time, without using the SD card, does it work for hours without hanging up or resettting ?

The reason I suggested reducing the number of calls to the file printing function, is that I don't know what time and memory space latency is involved in making all of those calls, and I don't know how much code they generate. Other people may know, but not me. I'd be inclined to write the whole output line into a single text line and then write it to the file once, that's just the way I would always do it, PaulS has a different view, well he would know more than me.

I've done some tests with latency time on SD cards, sometimes it is fast, sometimes it takes a long time, there is probably some reason for it to do with the buffering but from the point of view of the arduino sketch the latency can be pretty "random" or unpredictable. The point is, your code cannot be set up to "rely" on some particular performance.

There were some single quotes, but now I have changed all to double-quotes. Does this matter?

A single quote 'A' is used for only a single character. Any sort of sequence of more than one character requires double quotes "text".

I would have thought your previous code would have been a compiler error, I have no idea how it would have been interpreted. You can have a string with only one character "B", but you cannot fit more than one character into a single char variable.

I wish I could get rid of TinyGPS++ and SdFat libraries as well

If memory space really is your issue, the TinyGPS library still works fine and it is smaller than the TinyGPS++ library.

But your program runs, so it has enough space, at least to begin with. And then it may be running out later. So your issue seems to be, not that your program is too big, but that you have what is called a “memory leak”. You need to find it, and fix it. Making the program smaller without fixing the memory leak will only result in the outcome that the program crashes after ten minutes instead of five minutes.

I am still concerned that your hdop figure always seems to be zero.

Now, you seem to get data after about 6 seconds after your arduino appears to be re-setting. Do you have a backup battery on your GPS chip ? Because this 6 second delay looks like what the GPS chips call a hot start rather than a cold start. That may be some kind of clue.

You could check in your code, what would your code do if one of the NMEA sentences is missing or corrupted ? What would your code do if you "lost the gps fix" for two seconds and then regained it ? Would your code then try to start writing another brand new logging file ? That may be a logical flaw in your scheme.

Did you get rid of your three second delay in the loop ?

I had a look at the code of my GPS logging applications. I never try to open the same SD file which I had before. I have a system of numbered files and I always open a new file with a different number every time the process restarts. When I first started using SD files on the arduino about 3 years ago, I had a lot of reliability problems , and still do, although not as many.

The last suggestion, is to make either the GPS baud rate or the serial connection baud rate faster. The arduino then spends less time waiting around for serial data. I found that the whole GPS process works better at higher speeds.

One problem I had, was that the GPS chip was outputting 5 different messages, and sometimes it got corrupted and tried to write two of its messages simultaneously, which messed tham up. Hard to believe that the manufacturer did not think of that problem. I thought it was me, not reading them properly. Anyway, solved the problem by turning the ones I didn't need, off.

I use Mega's with multiple hardware serial. I am not an expert on software serial at all, I only used it once.