Go Down

Topic: [SOLVED] Problem data logging with TinyGPS++ and SdFat (Read 16299 times) previous topic - next topic

Sayedurrchowdhury

Dec 13, 2013, 02:11 pm Last Edit: Dec 16, 2013, 04:03 pm by Sayedurrchowdhury Reason: 1
A newbie here and my first ever post here. Please be supportive. And please bear with any silly mistakes I may have made in this post. So first thing first, here is my objective: I want to log GPS location and ancillary data at relatively low sample rate (say once every 3-5 seconds) on a MicroSD card.

And here is my hardware:

1. A strip board based  ATmega328p running at 8Mhz internal clock, powered by regulated 3.3V (otherwise running perfectly with most other sketches),

2. GPS breakout (I alternate between a uBlox Neo-6 from eBay, and Adafruit Ultimate GPS, basically no significant difference here, both perfectly spitting valid NMEA sentences and getting fix in short time)

3. A microSd breakout (which simply exposes MiscroSD pins to a header friendly 0.1" spaced pins, absolutely no other circuit involved)

4. An ordinary 2GB OEM MicroSd card formatted by SD formatter utility (otherwise perfectly working with another sketch at higher sampling/logging rates, i.e., combined DHT11 Humidity&Temp sensor + SR04 Ping sensor at 8-10 samples per second).

Now here is what my problem is:
SD card is detected, file opened, data logging starts - but cannot proceed as expected. Any of these things happen randomly- file content gets garbled (unexpected things including extended ASCII symbols get written, no consistent end-of-line, even some of my Serial.print stuff gets written here when I keep Serial output ON).

I assume, I have no hardware, electrical, electromagnetic, wiring, bad SD card, or wrong formatting issues based on consistent successes I have with other sketches. That brings me down to this particular GPS data logger sketch. It contains SoftwareSerial.h, TinyGPS++.h and SdFat.h. I know all three are well written, widely used, well tested and well run libraries, there shouldn't be any question.  So, my codes must have the root of the problem, unfortunately I cannot figure out what is the culprit.

I have checked available RAM at runtime and optimized RAM usage as much as I could. Now It shows FreeMem()=798 bytes in Setup(), and FreeMem()=792 bytes in Loop(). I have no clue what else I can do to solve my problem.

Here goes my code:
Code: [Select]

/*
GPS data logger using TinyGPS++ and SoftwareSerial libraries and examples by Mikal Hart
Sketch written by: Sayedur R Chowdhury, Dec 10, 2013

Data to be logged: Date, Time, Lat, Lon, Altitude, DOP, Speed, Bearing

Indicator LED:
  Uses one common anode RGB LED. Connections:
  Anode = pin 6
    Red   = pin 5
Green = pin 7
  Blue = pin 8
 
GPS connections:
  Rx = pin 2
  Tx = pin 3

MicroSD connections:
  CS = pin 10
    MOSI = pin 11
    MISO = pin 12
  SCK = pin 13

*/
//#define SERIALOUT

//#include <MemoryFree.h>
#include <TinyGPS++.h>
#include <SoftwareSerial.h>
#include <SdFat.h>

//definitions fo SoftwareSerial GPS
static const byte Rx = 3, Tx = 2;  //rx,tx (3,2) on the Arduino side connects to Tx, Rx of GPS respectively
static const int baudrate = 9600;

//defintion for RGB LED
static const byte commonAnode=6, R=5, G=7, B=8;

//definitions for SD card and file
SdFat sd;
SdFile datafile;
static const int SD_CS=10;

bool writeHeader=1;

//TinyGPS NMEA stream
TinyGPSPlus nmea;
//TinyGPSCustom pdop(nmea, "GPGSA", 15);  //takes up 50 bytes of SRAM

//serial connection to the GPS device
SoftwareSerial gps(Rx, Tx);


void setup()
{
#ifdef SERIALOUT
Serial.begin(9600);
#endif

//set up RGB led pins
pinMode(commonAnode, OUTPUT);
pinMode(R, OUTPUT);
pinMode(G, OUTPUT);
pinMode(B, OUTPUT);
digitalWrite(commonAnode, HIGH);
digitalWrite(R, HIGH);
digitalWrite(G, HIGH);
digitalWrite(B, HIGH);

gps.begin(baudrate);
pinMode(SD_CS, OUTPUT);

if (sd.begin(SD_CS, SPI_HALF_SPEED)) {
//indicate success by long flashing ok led twice
flashLED(1000, G);
smartDelay(500);
flashLED(1000, G);
#ifdef SERIALOUT
Serial.println(F("SD Ok"));
#endif
}
else {
//indicate SD init error by long flashing error-led once
flashLED(1000, R);
#ifdef SERIALOUT
Serial.println(F("SD Error"));
#endif
}  //SD.begin ends

#ifdef SERIALOUT
Serial.print("freeMemory()=");
Serial.println(freeMemory());
#endif
}    //setup ends

void loop()
{
unsigned int pastCharProcessed=nmea.charsProcessed(), recentCharProcessed=0;
int flashTimer=0;
long previousMillis = 0;
unsigned long currentMillis = millis();
//SdFile datafile;

//smartdelay at the beginning
smartDelay(3000);
//check new nmea.charProcessed because smartDelay has already elapsed some time
recentCharProcessed=nmea.charsProcessed()-pastCharProcessed;

flashTimer=currentMillis-previousMillis;

//verify if GPS data coming in
if (recentCharProcessed < 10) {    //not enough data coming in
//flash errorLed every 3 seconds if no data is coming, diagnose wiring problem
if (flashTimer > 3000) {
previousMillis = currentMillis;
flashLED(10, R);
flashTimer=0; //reset timer
#ifdef SERIALOUT
Serial.println(F('No data'));
#endif
}
}
else { //expected amount of data is coming
//verify if a fix is obtained
if (!nmea.location.isValid()) {  //no fix yet
//flash fixLed every 10 seconds if data is coming but no fix yet
if (flashTimer > 10000) {
previousMillis = currentMillis;
flashLED(10, B);
flashTimer=0; //reset timer
#ifdef SERIALOUT
Serial.println(F('Data Ok, no fix'));
#endif
}
}
else {  //fix obtained
//GPS units generally have fix indicators, an indicator is not needed; comment out flashLed codes below
/*
if (flashTimer > 30000) {
previousMillis = currentMillis;
flashLED(10, G);
flashTimer=0; //reset timer
}
*/
#ifdef SERIALOUT
Serial.println(F('Fix obtained'));
#endif

//Debug: the following closing brace moved here from below to initiate loggin even before a fix
//in the final sketch logging will start only when a fix is obtained
} //!nmea.location.isValid()

//now that a fix is obtained,  proceed onto logging data to SD card file

if (datafile.open("GPSDATA.CSV", O_WRITE | O_CREAT | O_APPEND)) {
//indicate file success by flashing OkLed once
//following conditional LED flashing will change into unconditional green in final sketch
if (nmea.location.isValid()) {
flashLEDbicolor(20, R, B);
} else {
  flashLED(20, G);
  }

#ifdef SERIALOUT
Serial.println(F("GPSDATA.CSV opened"));
#endif

//write CSV header only once at the beginning of the loop
if (writeHeader==1) {
datafile.println();
datafile.print(F("GPS data logging starts at "));
datafile.print(millis());
datafile.println(F(" mSecs"));
datafile.println(F("date,time,lat,lon,altitude,DOP,speed,course"));
writeHeader=0;
}

//date-time
if (nmea.date.isUpdated()) {
datafile.print(nmea.date.year());
datafile.print("/");
datafile.print(nmea.date.month());
datafile.print("/");
datafile.print(nmea.date.day());
}
datafile.print(",");
if (nmea.time.isUpdated()) {
datafile.print(nmea.time.hour());
datafile.print(":");
datafile.print(nmea.time.minute());
datafile.print(":");
datafile.print(nmea.time.second());
datafile.print(".");
datafile.print(nmea.time.centisecond());
}
datafile.print(",");

//location
datafile.print(nmea.location.lat(),6);
datafile.print(",");
datafile.print(nmea.location.lng(),6);
datafile.print(",");

//altitude
datafile.print(nmea.altitude.meters(),5);
datafile.print(",");

//DOP
datafile.print(nmea.hdop.value(),5);
datafile.print(",");

//speed
datafile.print(nmea.speed.kmph());
datafile.print(",");

//bearing
datafile.print(nmea.course.deg());

//last field saved, start a new line
datafile.println();

#ifdef SERIALOUT
Serial.println(F("Data written."));
#endif

//close datafile
datafile.close();
} //if datafile succeeds
else {  //datafile error

#ifdef SERIALOUT
Serial.println(F("Write error"));
#endif

//flash error LED
flashLED(400, R);
}  //datafile

//Debug: the following closing brace moved upward to intiate logging even before a fix
//} //!nmea.location.isValid() ends

} //recentCharProcessed < 10

#ifdef SERIALOUT
Serial.print("freeMemory()=");
Serial.println(freeMemory());
#endif
}

// This custom version of delay() ensures that the nmea object is being "fed".
static void smartDelay(unsigned long ms)
{
unsigned long start = millis();
do
{
while (gps.available())
nmea.encode(gps.read());
} while (millis() - start < ms);
}

//red LED indicator of Errors
static void flashLED(int duration, byte color)
{
digitalWrite(color, LOW);
smartDelay(duration);
digitalWrite(color, HIGH);
}

static void flashLEDbicolor(int duration, byte color1, byte color2)
{
digitalWrite(color1, LOW);
digitalWrite(color2, LOW);
smartDelay(duration);
digitalWrite(color1, HIGH);
digitalWrite(color2, HIGH);
}

Sayedurrchowdhury

#1
Dec 13, 2013, 03:25 pm Last Edit: Dec 13, 2013, 04:42 pm by Sayedurrchowdhury Reason: 1
UPDATE
--------

I have just had some success by turning OFF the Serial monitor output completely. Now data seems to be logging for some time. But as soon as the GPS gets position fix, it fails soon, not immediately but after writing a few records. Here is the latest output from my SD card:

Code: [Select]

GPS data logging starts at 5988 secs
date,time,lat,lon,altitude,DOP,speed,course
2000/0/0,0:0:0.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2000/0/0,14:23:46.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:49.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:52.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:55.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:23:58.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:1.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:4.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:7.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:10.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:13.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:16.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:19.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:22.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:25.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:28.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:31.0,0.000000,0.000000,304444,0.00000,0.00,0.00
2013/12/13,14:24:35.0,0.000000,0.000000,4114,0.00000,0.00,0.00
2013/12/13,14:24:38.0,22.340637,91.829925,4114,0.00000,2.72,193.13
2013/12/13,14:24:41.0,22.340641,91.829933,4114,0.00000,0.63,190.74
2013/12/13,14:24:44.0,22.340642,91.829933,4114,0.00000,0.74,191.47
2013/12/13,14:24:47.0,22.340648,91.829933,4114,0.00000,1.26,191.35
2013/12/13,14:24:50.0,22.340652,91.829933,4114,0.00000,1.02,191.02
2013/12/13,14:24:53.0,22.340654,91.829933,4114,0.00000,0.91,191.56


Sayedurrchowdhury

Well, I may have forgotten to mention that I needed some guidance/expert advice/help/clues - whatever you want to call it; here I am in need of some help. Please!

christian1328

Have you checked the SRAM according to this: http://playground.arduino.cc/Main/PROGMEM
For me it sounds like a RAM issue...

Cheers, Christian

Sayedurrchowdhury

@Christian

Thanks for reply. I have checked RAM by the following code ...

Code: [Select]

#include <MemoryFree.h>
...
...
#ifdef SERIALOUT
Serial.print("freeMemory()=");
Serial.println(freeMemory());
#endif

michinyon

if you send the GPS output to your computer instead,  can you see it ?

Are you getting gps readings,   gps doesn't work very well indoors.

The bottom part of the sd file example you show,  looks OK.  You are in India.  The first part is zeros,  because it takes the GPS several seconds or minutes to start. 

michinyon

Actually, I should be paying more attention.   You're in Chittagong.

Do you know how GPS modules work ?  It takes a second or two for it to detect the first satellite.  That's when it starts telling you the time and date.   It then takes anything between 10 seconds and 10 minutes to download the satellite almanac  and find at least 3 or 4 satellites.  That is when you get a position fix.   There is a code for this in the NMEA output.

If you don't want those zeros in your file,   don't start storing data in your file until you get that fix.

michinyon

The other problem I can see,  is that 3 second delay at the beginning of loop().    If your GPS is sending you a string every second,   that 3 second delay is going to cause a traffic jam of some kind.    Try making that delay a lot smaller.

I am not too keen on all those datafile.print() function calls.   I'd use sprintf() to write the result into a char array  and then write the whole line to the datafile at once.

michinyon

The other thing you might want to look into,  is why does you altitude and hdop figure look like nonsense.  Is there a type conflict there ?

Sayedurrchowdhury

#9
Dec 15, 2013, 03:22 pm Last Edit: Dec 15, 2013, 03:32 pm by Sayedurrchowdhury Reason: 1

Actually, I should be paying more attention.   You're in Chittagong.

Do you know how GPS modules work ?  It takes a second or two for it to detect the first satellite.  That's when it starts telling you the time and date.   It then takes anything between 10 seconds and 10 minutes to download the satellite almanac  and find at least 3 or 4 satellites.  That is when you get a position fix.   There is a code for this in the NMEA output.

If you don't want those zeros in your file,   don't start storing data in your file until you get that fix.


Thanks michinyon. Yes, I am in Chittagong (Bangladesh), not in India :D. I think I know how a GPS works, because I have a relevant Masters degree and several PhD courses on GIS, Remote Sensing, Survey and GPS technologies. I am a professional in this field since 2002, and operated a dozens of kinds of GPS hardware from simple handheld to DGPS/RTK, to navigational, to USB, bluetooth... you name it. I have mentioned and is apparent in my code that for debugging purpose I have initiated data logging before the GPS hardware gets a fix.

Thanks anyways, but I am still in dark as to why the MCU gets crazy after some time. RAM is a suspect, I am expecting an an expert to tell me whether 792 bytes of free RAM should be considered adequate, marginal or inadequate when 512 bytes of SD blocks has to be written and when these three libraries are in action.

Sayedurrchowdhury


The other thing you might want to look into,  is why does you altitude and hdop figure look like nonsense.  Is there a type conflict there ?


No it's because I had some mistake in my code (particularly the order of the fields in header row vs. the order of the fields in actual data rows), which I have now fixed.

Sayedurrchowdhury

With a little bit of minor tweaking here and there, and getting the device to start logging only after fix I have this output....

Code: [Select]

GPS data logging starts at 6516 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:19.0,22.340890,91.830078,0.00000,0,0.17,0.00
2013/12/13,17:13:22.0,22.340890,91.830078,0.00000,0,0.02,0.00

GPS data logging starts at 5826 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:31.0,22.340883,91.830070,0.00000,0,0.13,0.00
2013/12/13,17:13:34.0,22.340881,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:13:37.0,22.340879,91.830070,0.00000,0,0.02,0.00
2013/12/13,17:13:39.0,22.340879,91.830070,0.00000,0,0.02,0.00
2013/12/13,17:13:42.0,22.340877,91.830070,0.00000,0,0.13,0.00

GPS data logging starts at 5711 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:13:48.0,22.340875,91.830062,0.00000,0,0.09,0.00
2013/12/13,17:13:51.0,22.340873,91.830062,0.00000,0,0.06,0.00
2013/12/13,17:13:54.0,22.340873,91.830062,0.00000,0,0.13,0.00
2013/12/13,17:13:57.0,22.340873,91.830062,0.00000,0,0.04,0.00
2013/12/13,17:14:0.0,22.340873,91.830062,0.00000,0,0.04,0.00

GPS data logging starts at 5920 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:15.0,22.340877,91.830055,0.00000,0,0.07,0.00
2013/12/13,17:14:18.0,22.340877,91.830055,0.00000,0,0.11,0.00

GPS data logging starts at 5818 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:27.0,22.340877,91.830055,0.00000,0,0.07,0.00

GPS data logging starts at 5863 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:36.0,22.340879,91.830062,0.00000,0,0.20,0.00

GPS data logging starts at 5908 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:14:51.0,22.340885,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:14:54.0,22.340887,91.830070,0.00000,0,0.11,0.00
2013/12/13,17:14:57.0,22.340887,91.830070,0.00000,0,0.04,0.00
2013/12/13,17:15:0.0,22.340888,91.830070,0.00000,0,0.00,0.00
2013/12/13,17:15:3.0,22.340888,91.830062,0.00000,0,0.13,0.00

GPS data logging starts at 5918 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:15:19.0,22.340883,91.830070,0.00000,0,0.07,0.00

GPS data logging starts at 5898 mSecs
date,time,lat,lon,altitude,HDOP,speed,course
2013/12/13,17:15:25.0,22.340879,91.830070,0.00000,0,0.07,0.00
2013/12/13,17:15:28.0,22.340877,91.830070,0.00000,0,0.06,0.00
2013/12/13,17:15:31.0,22.340877,91.830078,0.00000,0,0.07,0.00



than means the MCU is reseting randomly after every few records... please someone give me some guidance which direction I should take now.

PaulS

Quote
I am not too keen on all those datafile.print() function calls.   I'd use sprintf() to write the result into a char array  and then write the whole line to the datafile at once.

SD data is buffered. So, writing one piece of information at a time to the SD buffer is better than writing to another buffer and then having that buffer added to the SD buffer.

Code: [Select]
long previousMillis = 0;
unsigned long currentMillis = millis();

Are you planning to support the concept of time going backwards? If not., all time variables are unsigned.

Code: [Select]
Serial.println(F('Fix obtained'));
Which ONE key did you press to get that ONE character between the single quotes?

Have you fixed the issue with the overly large value in the call to smartDelay()?

Sayedurrchowdhury

Though I am not getting advice to the level I'd expected, I continue to share my experience in a hope that someone like me may be benefited.

I just got rid of the SoftwareSerial Library in my code and that saves additional  112 bytes of precious RAM, may be even more. This time I am saving FreeMemory() return value to SD card, because the hardware serial is occupied by the GPS. I don't know if I could save few more bytes if I stop writing FreeMemory() return value to SD card.

I wish I could get rid of TinyGPS++ and SdFat libraries as well, but those are at the heart of the operation. Does anyone know thinner alternatives to these libraries, or did someone benefit from subsetting the libraries?

Sayedurrchowdhury

#14
Dec 15, 2013, 04:44 pm Last Edit: Dec 15, 2013, 04:46 pm by Sayedurrchowdhury Reason: 1
Thanks a lot Paul.

Quote
I am not too keen on all those datafile.print() function calls.   I'd use sprintf() to write the result into a char array  and then write the whole line to the datafile at once.

I am actually not a C person, I tried to concatenate all the fields in one string, but those pointer-char-string related compiler errors compelled me to back out. Any help (code snippet) would be highly appreciated.  But then, do you think that could improve my situation? Why and how? (Knowing the theory is always good :) )

Quote
Are you planning to support the concept of time going backwards? If not., all time variables are unsigned.

No I'm not. I shall modify if that helps. :)

Quote
Which ONE key did you press to get that ONE character between the single quotes?

There were some single quotes, but now I have changed all to double-quotes. Does this matter?

Quote
Have you fixed the issue with the overly large value in the call to smartDelay()?

I thought if I use long delay in smartDelay(), old values would be overwritten by new values that are constantly coming from GPS. Doesn't TinyGPS++ take care of that?

Go Up