Go Down

Topic: [SOLVED] Problem data logging with TinyGPS++ and SdFat (Read 16298 times) previous topic - next topic

michinyon

I'm sorry if my first advice seemed odd.  I was looking at the first section of output you printed,   and it looked like all zeros,   I didn't see there was some actual data down at the bottom of the file,   it was out of the viewing window.

You did not say you were a GPS expert.

If your program runs and then hangs up, or restarts,  then it is likely to be one of three things.   Program corruption ( unlikely on the arduino ) ,  running out of memory ( much more likely ),  or some kind of power supply problem.   How reliable is your arduino power supply ?   Is there enough power to keep your GPS chip running ?

This TinyGPSPlus is new,  I didn't see it before today,  I just downloaded it and was playing with it myself.  Are you using the very latest version of it ?   Are you sure there is no memory leak in it?    Try running it for a long time,  without using the SD card,    does it work for hours without hanging up or resettting ?

The reason I suggested reducing the number of calls to the file printing function,   is that I don't know what time and memory space latency is involved in making all of those calls,   and I don't know how much code they generate.   Other people may know,  but not me.   I'd be inclined to write the whole output line into a single text line    and then write it to the file once,   that's just the way I would always do it,   PaulS has a different view,   well he would know more than me.

I've done some tests with latency time on SD cards,   sometimes it is fast,   sometimes it takes a long time,   there is probably some reason for it to do with the buffering but from the point of view of the arduino sketch the latency can be pretty "random" or unpredictable.    The point is,   your code cannot be set up to "rely" on some particular performance.





michinyon

Quote
There were some single quotes, but now I have changed all to double-quotes. Does this matter?


A single quote  'A'  is used for only a single character.     Any sort of sequence of more than one character requires double quotes "text".

I would have thought your previous code would have been a compiler error,   I have no idea how it would have been interpreted.    You can have a string with only one character  "B",    but you cannot fit more than one character into a single  char variable.

michinyon

Quote
I wish I could get rid of TinyGPS++ and SdFat libraries as well


If memory space really is your issue,    the TinyGPS  library still works  fine  and it is smaller than the TinyGPS++  library.

But your program runs,    so it has enough space,  at least to begin with.   And then it may be running out later.    So your issue seems to be,  not that your program is too big,    but that you have what is called a "memory leak".   You need to find it, and fix it.    Making the program smaller  without fixing the memory leak  will only result in the outcome that the program crashes after ten minutes instead of five minutes.






michinyon

I am still concerned that your hdop figure always seems to be zero.

Now,   you seem to get data after about 6 seconds after your arduino appears to be re-setting.   Do you have a backup battery on your GPS chip  ?    Because this 6 second delay looks like what the GPS chips call a hot start   rather than a cold start.  That may be some kind of clue.

You could check in your code,    what would your code do  if one of the NMEA sentences is missing or corrupted ?   What would your code do if you "lost the gps fix"  for two seconds and then regained it ?   Would your code then try to start writing another brand new logging file ?   That may be a logical flaw in your scheme.

Did you get rid of your three second delay in the loop ?

I had a look at the code of my GPS logging applications.     I never try to open the same SD file which I had before.   I have a system of numbered files  and I always open a new file with a different number every time the process restarts.     When I first started using SD files on the arduino about 3 years ago,   I had a lot of reliability problems ,   and still do,  although not as many. 


michinyon

The last suggestion,   is to make either the GPS baud rate or the serial connection baud rate faster.      The arduino then spends less time waiting around for serial data.   I found that the whole GPS process works better at higher speeds.

One problem I had,  was that the GPS chip was outputting 5 different messages,    and sometimes it got corrupted and tried to write two of its messages simultaneously,  which messed tham up.   Hard to believe that the manufacturer did not think of that problem.   I thought it was me,  not reading them properly.  Anyway,  solved the problem by turning the ones I didn't need, off.

I use Mega's with multiple hardware serial.    I am not an expert on software serial at all, I only used it once.

Sayedurrchowdhury


I'm sorry if my first advice seemed odd.  I was looking at the first section of output you printed,   and it looked like all zeros,   I didn't see there was some actual data down at the bottom of the file,   it was out of the viewing window.

You did not say you were a GPS expert.


Oops! seems like I have replied a bit harsh, sorry about that. And I am not a GPS expert per se, but I have been taught GPS technology at schools; I do use them a lot, of different varieties in different situations for different purposes.

And then thank you so much for giving some real food for thought and work.

Quote
If your program runs and then hangs up, or restarts,  then it is likely to be one of three things.   Program corruption ( unlikely on the arduino ) ,  running out of memory ( much more likely ),  or some kind of power supply problem.   How reliable is your arduino power supply ?   Is there enough power to keep your GPS chip running ?


Hmm, you give me another suspect, I am gonna put the power supply under the microscope now. One question though, when you say enough power do you mean voltage or current? I am running the chip at 3.3V (actually getting 3.2V) from a 9V Alkaline battery, stepped down in two steps by LM7805 and then by LM371 voltage regulator chips. This is exactly what I have made for myself: http://www.instructables.com/id/Dual-voltage-regulated-power-supply-for-Arduinomic/

Quote
This TinyGPSPlus is new,  I didn't see it before today,  I just downloaded it and was playing with it myself.  Are you using the very latest version of it ?   Are you sure there is no memory leak in it?    Try running it for a long time,  without using the SD card,    does it work for hours without hanging up or resettting ?


Yes, TinyGPS++ is relatively new, and I have downloaded this week, should be the latest build. I have mentioned at the very first line of the first post that I am a 'newbie' in Arduino field, and mentioned somewhere else that I know almost no C. I don't even understand what is Memory Leak (sorry my bad). I would highly appreciate any easy going online reading material / link. Once I get the idea, I will put that under microscope too. :D

I will put the assembly to run for long time with and without SD card - you suspect anything there? Let's see if I discover something that I can understand and interpret.

Quote
The reason I suggested reducing the number of calls to the file printing function,   is that I don't know what time and memory space latency is involved in making all of those calls,   and I don't know how much code they generate.   Other people may know,  but not me.   I'd be inclined to write the whole output line into a single text line    and then write it to the file once,   that's just the way I would always do it,   PaulS has a different view,   well he would know more than me.


At the beginning I tried to put everything together in a buffer and then write at once, but not knowing the C language is my weakness, I couldn't get past the compiler errors relating to strings and pointers and what not. :p

Quote
I've done some tests with latency time on SD cards,   sometimes it is fast,   sometimes it takes a long time,   there is probably some reason for it to do with the buffering but from the point of view of the arduino sketch the latency can be pretty "random" or unpredictable.    The point is,   your code cannot be set up to "rely" on some particular performance.


hmm, noted. I but think many other data logging applications around are still running with quite satisfactory reliability and performance, why not mine? I must have problems with my coding. I want to learn where, and how to fix - that's the whole point of this call for help. Thanks indeed for all these insightful responses.

Sayedurrchowdhury


A single quote  'A'  is used for only a single character.     Any sort of sequence of more than one character requires double quotes "text".

I would have thought your previous code would have been a compiler error,   I have no idea how it would have been interpreted.    You can have a string with only one character  "B",    but you cannot fit more than one character into a single  char variable.


As I told before, the programming language is one of my weaknesses, and these silly things tell you that  :P

Sayedurrchowdhury

#22
Dec 16, 2013, 12:38 am Last Edit: Dec 16, 2013, 12:49 pm by Sayedurrchowdhury Reason: 1

If memory space really is your issue,    the TinyGPS  library still works  fine  and it is smaller than the TinyGPS++  library.

I'll give the earlier TinyGPS a try.

Quote
But your program runs,    so it has enough space,  at least to begin with.   And then it may be running out later.

Memory doesn't seem to run outer later. I dumped FreeMemory() to SD card at the end of each GPS data record, and that reads constant 792 bytes free with SoftwareSerial and 904 bytes free without it after  every pass of the loop.

Quote
But that you have what is called a "memory leak".   You need to find it, and fix it.  

Again, I'll have to read about memory leak before I can address it. Let's see if I can find some good read.

Thanks again, michinyon!!

Sayedurrchowdhury

#23
Dec 16, 2013, 01:27 am Last Edit: Dec 16, 2013, 12:54 pm by Sayedurrchowdhury Reason: 1

I am still concerned that your hdop figure always seems to be zero.

I'll have to check it again, I am guessing for now that the GPS itself is not providing DOP data. Some GPSs do use their own variant of NMEA sentences - somewhat scaled down.

Quote
Now,   you seem to get data after about 6 seconds after your arduino appears to be re-setting.   Do you have a backup battery on your GPS chip  ?    Because this 6 second delay looks like what the GPS chips call a hot start   rather than a cold start.  That may be some kind of clue.

Yes, here the GPS is taking about 5-6 secs to hot start. I have not yet attached the RTC battery to the GPS module. But how could possibly hot/warm/cold start (and the associated delay) be linked to resetting of the chip? Please give more clue I can look into.  

Quote
what would your code do  if one of the NMEA sentences is missing or corrupted ?   What would your code do if you "lost the gps fix"  for two seconds and then regained it ?   Would your code then try to start writing another brand new logging file ?   That may be a logical flaw in your scheme.

I think TinyGPS++ itself checks which NMEA sentence is complete and valid, and which is not (missing, corrupt, or incomplete). Based on valid and complete sentences it gives out parsed data. I assume (can't figure out from library codes) that  the library updates the data only based on the valid sentences, and the sketch gets only the latest data values, older data is just overwritten in the library. Correct me if I am wrong (I know you have seen the library code by now).

If my GPS lose fix, my code should just loop though the loop() without writing anything to SD, and resume writing as soon a new fix is obtained. But that depends on how the library  handles the situations. I will have to check what the library returns when the GPS lose fix and I call gps.location.isValid() and gps.location.lat() etc. Is it the the old value calculated earlier when there was a fix? or is it a 'No valid location'? - I am not sure, have to check.

Quote
Did you get rid of your three second delay in the loop ?

No, I didn't, coz I suppose the library constantly overwrites return values with newly arrived NMEA data. So it doesn't matter how long I wait (by smartDelay, not usual delay) before calling a data returning function. Am I deadly wrong?

Quote
I had a look at the code of my GPS logging applications.     I never try to open the same SD file which I had before.   I have a system of numbered files  and I always open a new file with a different number every time the process restarts.     When I first started using SD files on the arduino about 3 years ago,   I had a lot of reliability problems ,   and still do,  although not as many.  

I saw that incremental numbering of file names in Adafruit library examples, I will perhaps use that scheme at the mature stage of my application, but now that my Arduino is resetting too frequently, having that scheme will be painful now, there will be too many files to keep track of and examine.

Sayedurrchowdhury

#24
Dec 16, 2013, 01:41 am Last Edit: Dec 16, 2013, 01:00 pm by Sayedurrchowdhury Reason: 1

The last suggestion,   is to make either the GPS baud rate or the serial connection baud rate faster.      The arduino then spends less time waiting around for serial data.   I found that the whole GPS process works better at higher speeds.

Good point, I will try different baud rates to see if that affects performance.

Quote
One problem I had,  was that the GPS chip was outputting 5 different messages,    and sometimes it got corrupted and tried to write two of its messages simultaneously,  which messed tham up.   Hard to believe that the manufacturer did not think of that problem.   I thought it was me,  not reading them properly.  Anyway,  solved the problem by turning the ones I didn't need, off.

Depending on which GPS hardware you use and how you configure it, it can even send you 15 different messages. 4 ot 5 are most common, e.g. GPGSA, GPGSV, GPRMC, etc. But with a vast majority of modern GPS chips you can control at least three things: 1) which sentences you want it to output, 2) at what frequency it samples internally, 3) at what frequency it outputs. So you have much better control of the amount of data you actually have to deal with, it is not necessarily 5 sentences every second. I have mine configured to output only GPRMC and GPGGA sentences once every 2 seconds.

Hey, great having your valuable suggestions. Thanks once again!

mikalhart

Hi all--

I agree with the suggestion to increase the serial console baud rate to as large as can be supported.  I always use 115200 baud.  If you print to the console at slow baud rates you unnecessarily increase the risk that GPS data will be dropped.  That won't explain the device resets, but it would explain not being able to get a TinyGPS++ fix.

I wouldn't increase the GPS baud rate unless your application requires it.

I don't immediately see an issue in the code that would cause the resets, but I doubt it's in the TinyGPS++ library.  Yes, memory leakage can certainly cause unexpected resets, but TinyGPS++ doesn't use dynamic allocation, and field tests suggest that the library doesn't have any stack space leakage problems.  We know of several groups using TinyGPS++ in weather balloon logging applications.  Lat/long/altitude, etc., are logged to SD card every second over the course of many, many hours.

Could it be a wiring issue?




Sayedurrchowdhury

#26
Dec 16, 2013, 02:40 pm Last Edit: Dec 16, 2013, 04:22 pm by Sayedurrchowdhury Reason: 1

I agree with the suggestion to increase the serial console baud rate to as large as can be supported.  I always use 115200 baud.  If you print to the console at slow baud rates you unnecessarily increase the risk that GPS data will be dropped.  That won't explain the device resets, but it would explain not being able to get a TinyGPS++ fix.

Hi Mikal, Thanks for your feedback.

In my final application I am not using Serial console at all (#define SERIALOUT conditional compiler directive deactivated). So that's not an issue. GPS baud rate also must not be an issue, because I have set my GPS to send only GPRMC and GPGGA once every two seconds, i.e. less than 500-600 bits per second. I agree that TinyGPS++ should be fine with generating a location values as long as it gets valid NMEA sentences with location information. I have checked the GPS device hooking with my computer and software, it is generating valid NMEA sentences with fix.

Quote
I wouldn't increase the GPS baud rate unless your application requires it.

I shall go with that, I don't need high data rate, my sampling frequency is less than 0.35 Hz.

Quote
Could it be a wiring issue?

Yes I am starting to suspect, and paying more attention to that.

Thanks a lot.

Sayedurrchowdhury

#27
Dec 16, 2013, 03:58 pm Last Edit: Dec 16, 2013, 04:02 pm by Sayedurrchowdhury Reason: 1
I think I have got to the bottom of the problem, at least nearly so. I have diagnosed that it was the worn out battery that was causing the chip to reset. I was using this battery for quite some time, and it was nearly out of any real power. As soon as I replaced the device with a new 9V battery of the same kind, it seems to be working as it should. I have just run a half hour logging, and it did not reset.

So the last known good configuration is as follows:
1. Battery in good condition.
2. SoftwareSerial library disengaged.
3. GPS device read using hardware serial at 19200 baud.
4. NMEA output configuration: All sentences at 1 Hz rate
5. ATmega328p, SD and GPS powered at 3.3V

Final thoughts:

1) The reset was perhaps being caused by what is called 'brown-out'. I am running the chip at 3.3V, but not sure enough whether its fuses are properly set to operate at lower voltage. Any help from experts regarding lowering the brown-out detection level would be much appreciated.

2) FreeMemory() dump on SD card shows constant maintenance of 902 bytes, suggesting that there is perhaps no memory leak. Or isn't there really?

And here is a small section from SD file data dump
Code: [Select]

GPS data logging starts at 5994 mSecs
date,time,lat,lon,altitude,HDOP,speed,course,FreeMem
2013/12/16,14:23:35.0,22.340959,91.830001,0.30000,1300,1.41,0.00,902
2013/12/16,14:23:39.0,22.340953,91.830001,1.60000,1300,1.39,254.40,902
2013/12/16,14:23:41.0,22.340957,91.830001,1.10000,1300,0.33,254.40,902
2013/12/16,14:23:45.0,22.340965,91.830009,1.00000,1300,0.22,254.40,902
2013/12/16,14:23:47.0,22.340965,91.830009,1.00000,304444,0.22,254.40,902
2013/12/16,14:23:51.0,22.340965,91.830009,1.00000,304444,0.22,254.40,902

... several hundred records deleted here .....

2013/12/16,14:48:40.0,22.341001,91.829986,20.10000,343,0.67,37.88,902
2013/12/16,14:48:42.0,22.340993,91.829986,18.70000,401,0.41,37.88,902
2013/12/16,14:48:46.0,22.340986,91.829978,16.70000,343,0.13,37.88,902
2013/12/16,14:48:48.0,22.340978,91.829978,15.70000,343,0.41,37.88,902
2013/12/16,14:48:52.0,22.340969,91.829971,13.60000,343,0.30,37.88,902
2013/12/16,14:48:56.0,22.340965,91.829963,13.00000,343,0.39,37.88,902

mikalhart

That's great news, Sayedur.  I should have thought of the low battery problem possibility.

Just FYI, when we do logging like this we use AA or AAA batteries, which have much greater capacities (and therefore lifespans) than 9V.  In your case I would probably use 3 batteries regulated down to 3.1V.  (We use 2 or 3 boosted UP to 5V.)

I don't think there is any memory leak.


Sayedurrchowdhury


That's great news, Sayedur.  I should have thought of the low battery problem possibility.

Just FYI, when we do logging like this we use AA or AAA batteries, which have much greater capacities (and therefore lifespans) than 9V.  In your case I would probably use 3 batteries regulated down to 3.1V.  (We use 2 or 3 boosted UP to 5V.)

I don't think there is any memory leak.




Thanks Mikal, with a little hardware modification I can arrange to have other types of batteries connected to the device. I was considering using 3.7V LiPo for field deployment of the device.

Go Up