Making GPS + Accel + Distance data logger

Hello all,

First off, I am not a programmer by nature. I am a mechanical engineer and I usually just stumble my way through programming when needed. I am currently working on a project and I have completely exhausted my already weak programming skills. The project is this: I want to make a datalogger that records GPS data, accelerometer data, and distance data (Ping ultrasonic sensor) at 10hz(ish) to an SD card.

I am using the ping ultrasonic distance sensor, SD card shield, Sparkfun Venus GPS module, and sparkfun triaxial accel.

Where I am so far:

It was very easy to get the distance sensor and accelerometer to record to the SD card at 10hz. No issues with those. I am having trouble implementing the GPS. When I add code to log the GPS coordinates, everything compiles correctly, but nothing happens either on the serial monitor or the SD card. When I remove the SD logging code and keep the GPS code, everything works on the serial monitor but obviously there is no logging. (The timing is a bit off however).

Here is the code with the SD logging stuff commented out. This allows all functions to be viewed in the serial monitor (with screwy timing) but obviously nothing is saved to the SD card

#include <SD.h>
#include <Wire.h>

#include <TinyGPS++.h>
#include <SoftwareSerial.h>

// A simple data logger for the Arduino analog pins

// how many milliseconds between grabbing data and logging it. 1000 ms is once a second
// mills between entries (reduce to take more/faster data)

// how many milliseconds before writing the logged data permanently to disk
// set it to the LOG_INTERVAL to write each time (safest)
// set it to 10*LOG_INTERVAL to write all data every 10 datareads, you could lose up to 
// the last 10 reads if power is lost but it uses less power and is much f // mills between calls to flush() - to write data to the card
uint32_t syncTime = 0; // time of last sync()

#define ECHO_TO_SERIAL   1 // echo data to serial port
#define WAIT_TO_START    0 // Wait for serial input in setup()

// the digital pins that connect to the LEDs
#define redLEDpin 2
#define greenLEDpin 3

// The analog pins that connect to the sensors
#define AccelX 0                // analog 0
#define AccelY 1                // analog 1
#define BANDGAPREF 14            // special indicator that we want to measure the bandgap

#define aref_voltage 3.3         // we tie 3.3V to ARef and measure it with a multimeter!
#define bandgap_voltage 1.1      // this is not super guaranteed but its not -too- off



// for the data logging shield, we use digital pin 10 for the SD cs line
const int chipSelect = 10;
const int trigPin = 7;
const int echoPin = 6;
static const int RXPin = 3, TXPin = 2;
static const uint32_t GPSBaud = 9600;

TinyGPSPlus gps;

SoftwareSerial ss(RXPin, TXPin);

// the logging file
File logfile;

void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  
  // red LED indicates error
  digitalWrite(redLEDpin, HIGH);

  while(1);
}

void setup()
{
  Serial.begin(9600);
  ss.begin(GPSBaud);
 
 /* 

  Serial.println();
  
  // use debugging LEDs
  pinMode(redLEDpin, OUTPUT);
  pinMode(greenLEDpin, OUTPUT);
  


  // initialize the SD card
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    error("Card failed, or not present");
  }
  Serial.println("card initialized.");
  
  // create a new file
  char filename[] = "LOGGER00.CSV";
  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = i/10 + '0';
    filename[7] = i%10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE); 
      break;  // leave the loop!
    }
  }
  
  if (! logfile) {
    error("couldnt create file");
  }
  
  Serial.print("Logging to: ");
  Serial.println(filename);
 logfile.println("Date,Time,Accel X,Accel Y,Fork Travel,Latitude,Longitude,Speed");
*/
}
//______________________________________________________________________________


void loop()
{
   while (ss.available() > 0)
    if (gps.encode(ss.read()))
    {

     
  
  long duration, inches, mm;
  
  pinMode(trigPin, OUTPUT);
  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);
  
  pinMode(echoPin, INPUT);
  duration = pulseIn(echoPin, HIGH);
  mm = microsecondsToMillimeters(duration);
    // read the input on analog pin 0:
  int accelX = analogRead(A0);
  int accelY = analogRead(A1);
  // Convert the analog reading (which goes from 0 - 1023) to a voltage (0 - 5V):
  float gX = (accelX * (3.3 / 1023.0) - 1.08) /.22;
  float gY = (accelY * (3.3 / 1023.0) -1.08) / .22;

  // print out the value you read:
  //printDateTime(gps.date, gps.time);
  
//SERIAL----------------------------------------------------  

  //date
  if (gps.date.isValid())
  {
  Serial.print(gps.date.month());
  Serial.print("/");
  Serial.print(gps.date.day());
  Serial.print("/");
  Serial.print(gps.date.year());
  Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  
  //Time
  if (gps.time.isValid())
  {
    if (gps.time.hour() < 10) Serial.print(F("0"));
    Serial.print(gps.time.hour());
    Serial.print(F(":"));
    if (gps.time.minute() < 10) Serial.print(F("0"));
    Serial.print(gps.time.minute());
    Serial.print(F(":"));
    if (gps.time.second() < 10) Serial.print(F("0"));
    Serial.print(gps.time.second());
    Serial.print(F("."));
    if (gps.time.centisecond() < 10) Serial.print(F("0"));
    Serial.print(gps.time.centisecond());
    Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  
  //Accel
  Serial.print(gX);
  Serial.print(",");
  Serial.print(gY);
  Serial.print(",");
  
  //distance
  Serial.print(mm);
  Serial.print(",");
  
  //lat long
  if (gps.location.isValid())
  {
  Serial.print(gps.location.lat(), 6);
  Serial.print(",");
  Serial.print(gps.location.lng(), 6);
  Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  //speed
  if (gps.speed.isValid())
  {
  Serial.print(gps.speed.mph(), 6);
  Serial.println(",");
  }
  else
  {
    Serial.println(F("INVALID,"));
  }

//LOGFILE------------------------------------------------------------------
/*

  //date
  logfile.print(gps.date.month());
  logfile.print("/");
  logfile.print(gps.date.day());
  logfile.print("/");
  logfile.print(gps.date.year());
  logfile.print(",");
  
  //time
    if (gps.time.hour() < 10) logfile.print(F("0"));
    logfile.print(gps.time.hour());
    logfile.print(F(":"));
    if (gps.time.minute() < 10) logfile.print(F("0"));
    logfile.print(gps.time.minute());
    logfile.print(F(":"));
    if (gps.time.second() < 10) logfile.print(F("0"));
    logfile.print(gps.time.second());
    logfile.print(F("."));
    if (gps.time.centisecond() < 10) logfile.print(F("0"));
    logfile.print(gps.time.centisecond());
    logfile.print(",");
  
  //accel
  logfile.print(gX);
  logfile.print(",");
  logfile.print(gY);
  logfile.print(",");
  
  //distance
  logfile.print(mm);
  logfile.print(",");
  
  //lat long
  logfile.print(gps.location.lat(), 6);
  logfile.print(",");
  logfile.print(gps.location.lng(), 6);
  logfile.print(",");
  
  //speed
  logfile.print(gps.speed.mph());
  logfile.println(",");


 

 
  
  digitalWrite(greenLEDpin, LOW);

  // Now we write data to disk! Don't sync too often - requires 2048 bytes of I/O to SD card
  // which uses a bunch of power and takes time 
  // blink LED to show we are syncing data to the card & updating FAT!
  digitalWrite(redLEDpin, HIGH);
  logfile.flush();
  digitalWrite(redLEDpin, LOW);
  */
  

  }
}
 
 


long microsecondsToMillimeters(long microseconds)
{return microseconds / 2.9 / 2;
}

This code is what I feel like it should be to make everything work, but for some reason nothing happens.

#include <SD.h>
#include <Wire.h>
#include "RTClib.h"
#include <TinyGPS++.h>
#include <SoftwareSerial.h>

// A simple data logger for the Arduino analog pins

// how many milliseconds between grabbing data and logging it. 1000 ms is once a second
// mills between entries (reduce to take more/faster data)

// how many milliseconds before writing the logged data permanently to disk
// set it to the LOG_INTERVAL to write each time (safest)
// set it to 10*LOG_INTERVAL to write all data every 10 datareads, you could lose up to 
// the last 10 reads if power is lost but it uses less power and is much f // mills between calls to flush() - to write data to the card
uint32_t syncTime = 0; // time of last sync()

#define ECHO_TO_SERIAL   1 // echo data to serial port
#define WAIT_TO_START    0 // Wait for serial input in setup()

// the digital pins that connect to the LEDs
#define redLEDpin 2
#define greenLEDpin 3

// The analog pins that connect to the sensors
#define AccelX 0                // analog 0
#define AccelY 1                // analog 1
#define BANDGAPREF 14            // special indicator that we want to measure the bandgap

#define aref_voltage 3.3         // we tie 3.3V to ARef and measure it with a multimeter!
#define bandgap_voltage 1.1      // this is not super guaranteed but its not -too- off



// for the data logging shield, we use digital pin 10 for the SD cs line
const int chipSelect = 10;
const int trigPin = 7;
const int echoPin = 6;
static const int RXPin = 3, TXPin = 2;
static const uint32_t GPSBaud = 9600;

TinyGPSPlus gps;

SoftwareSerial ss(RXPin, TXPin);

// the logging file
File logfile;

void error(char *str)
{
  Serial.print("error: ");
  Serial.println(str);
  
  // red LED indicates error
  digitalWrite(redLEDpin, HIGH);

  while(1);
}

void setup(void)
{
  Serial.begin(9600);
  ss.begin(GPSBaud);
  Serial.println();
  
  // use debugging LEDs
  pinMode(redLEDpin, OUTPUT);
  pinMode(greenLEDpin, OUTPUT);
  


  // initialize the SD card
  Serial.print("Initializing SD card...");
  // make sure that the default chip select pin is set to
  // output, even if you don't use it:
  pinMode(10, OUTPUT);
  
  // see if the card is present and can be initialized:
  if (!SD.begin(chipSelect)) {
    error("Card failed, or not present");
  }
  Serial.println("card initialized.");
  
  // create a new file
  char filename[] = "LOGGER00.CSV";
  for (uint8_t i = 0; i < 100; i++) {
    filename[6] = i/10 + '0';
    filename[7] = i%10 + '0';
    if (! SD.exists(filename)) {
      // only open a new file if it doesn't exist
      logfile = SD.open(filename, FILE_WRITE); 
      break;  // leave the loop!
    }
  }
  
  if (! logfile) {
    error("couldnt create file");
  }
  
  Serial.print("Logging to: ");
  Serial.println(filename);
 logfile.println("Date,Time,Accel X,Accel Y,Fork Travel,Latitude,Longitude,Speed");
}
//______________________________________________________________________________


void loop(void)
{
 
  long duration, inches, mm;
  
  pinMode(trigPin, OUTPUT);
  digitalWrite(trigPin, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin, LOW);
  
  pinMode(echoPin, INPUT);
  duration = pulseIn(echoPin, HIGH);
  mm = microsecondsToMillimeters(duration);
    // read the input on analog pin 0:
  int accelX = analogRead(A0);
  int accelY = analogRead(A1);
  // Convert the analog reading (which goes from 0 - 1023) to a voltage (0 - 5V):
  float gX = (accelX * (3.3 / 1023.0) - 1.08) /.22;
  float gY = (accelY * (3.3 / 1023.0) -1.08) / .22;

  // print out the value you read:
  //printDateTime(gps.date, gps.time);
  
//SERIAL----------------------------------------------------  

  //date
  if (gps.date.isValid())
  {
  Serial.print(gps.date.month());
  Serial.print("/");
  Serial.print(gps.date.day());
  Serial.print("/");
  Serial.print(gps.date.year());
  Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  
  //Time
  if (gps.time.isValid())
  {
    if (gps.time.hour() < 10) Serial.print(F("0"));
    Serial.print(gps.time.hour());
    Serial.print(F(":"));
    if (gps.time.minute() < 10) Serial.print(F("0"));
    Serial.print(gps.time.minute());
    Serial.print(F(":"));
    if (gps.time.second() < 10) Serial.print(F("0"));
    Serial.print(gps.time.second());
    Serial.print(F("."));
    if (gps.time.centisecond() < 10) Serial.print(F("0"));
    Serial.print(gps.time.centisecond());
    Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  
  //Accel
  Serial.print(gX);
  Serial.print(",");
  Serial.print(gY);
  Serial.print(",");
  
  //distance
  Serial.print(mm);
  Serial.print(",");
  
  //lat long
  if (gps.location.isValid())
  {
  Serial.print(gps.location.lat(), 6);
  Serial.print(",");
  Serial.print(gps.location.lng(), 6);
  Serial.print(",");
  }
  else
  {
    Serial.print(F("INVALID,"));
  }
  //speed
  if (gps.speed.isValid())
  {
  Serial.print(gps.speed.mph(), 6);
  Serial.println(",");
  }
  else
  {
    Serial.println(F("INVALID,"));
  }
  
//LOGFILE------------------------------------------------------------------

  //date
  logfile.print(gps.date.month());
  logfile.print("/");
  logfile.print(gps.date.day());
  logfile.print("/");
  logfile.print(gps.date.year());
  logfile.print(",");
  
  //time
    if (gps.time.hour() < 10) logfile.print(F("0"));
    logfile.print(gps.time.hour());
    logfile.print(F(":"));
    if (gps.time.minute() < 10) logfile.print(F("0"));
    logfile.print(gps.time.minute());
    logfile.print(F(":"));
    if (gps.time.second() < 10) logfile.print(F("0"));
    logfile.print(gps.time.second());
    logfile.print(F("."));
    if (gps.time.centisecond() < 10) logfile.print(F("0"));
    logfile.print(gps.time.centisecond());
    logfile.print(",");
  
  //accel
  logfile.print(gX);
  logfile.print(",");
  logfile.print(gY);
  logfile.print(",");
  
  //distance
  logfile.print(mm);
  logfile.print(",");
  
  //lat long
  logfile.print(gps.location.lat(), 6);
  logfile.print(",");
  logfile.print(gps.location.lng(), 6);
  logfile.print(",");
  
  //speed
  logfile.print(gps.speed.mph());
  logfile.println(",");


 

 
  
  digitalWrite(greenLEDpin, LOW);

  // Now we write data to disk! Don't sync too often - requires 2048 bytes of I/O to SD card
  // which uses a bunch of power and takes time 
  // blink LED to show we are syncing data to the card & updating FAT!
  digitalWrite(redLEDpin, HIGH);
  logfile.flush();
  digitalWrite(redLEDpin, LOW);
 // delay (100);
  
  

  
  
}

long microsecondsToMillimeters(long microseconds)
{return microseconds / 2.9 / 2;
}

Please forgive my very sloppy code - it is mostly just code I found for each piece of hardware combined together and modified.

Any help or advice anyone can give me to help me move forward with this project would be very appreciated! Let me know if I left out any crucial details.

Thanks in advance!

You may be running out of RAM memory. Memory use is dynamic, but you might learn something with this little function.

int freeRam () {
  extern int __heap_start, *__brkval; 
  int v; 
  return (int) &v - (__brkval == 0 ? (int) &__heap_start : (int) __brkval); 
}

You've used the F macro sporadically to move string literals used by serial.print to progmem. E.g.

    Serial.print(F(":"));

You could save some RAM by doing it elsewhere too. E.g.

  Serial.print("Logging to: ");

Although I can't say why nothing happens specifically, I can say why it doesn't work in general. I don't think you're running out of RAM, as you have used the F macro for most (but not all???) strings.

It is common for one part to work by itself, and it is very common for several parts to not work together. In isolation, code written for one part does not have to worry about using up resources (CPU time and RAM) that might be needed by other parts. Throw them all together and you may not have enough CPU time or RAM to do what you want.

In this case, SoftwareSerial probably gets overwhelmed by the GPS data because the CPU is also checking the accelerometer pulse width and writing data to the SD and writing debug info. As a result, gps.decode will never get a complete sentence. It may get 8 bytes, then lose 20 bytes, receive 8 more bytes, then the checksum fails and it all gets pitched.

pulseIn may also get interrupted while waiting for the pin to change. Incoming GPS data may make it miss the rising or falling edge of a pulse, which will cause invalid values. [

pulseIn

](http://arduino.cc/en/Reference/pulseIn) can wait for up to 3 minutes, so I would suggest passing a 3rd argument for the timeout value. Pass the maximum pulse width from the accelerometer.

Fundamentally, you will have to change the structure of the loop to handle things in a priority order:

1. Process GPS data. The GPS device will emit a bunch of data at the start of a one-second period. When all that data has been sent, it quits sending data until the next one-second period begins. This is a "quiet" time during which you can do other things, like...

2. Check the pulse width. If the pulses are on the order of 10-500uS, you can get it done fairly quickly. If the pulses are > 1mS, you should not use pulseIn. I would also suggest writing the integer analog value instead of doing floating-point calculations in the Arduino. Whatever looks at the SD data can do that calculation for you (i.e., in the PC spreadsheet). If you are just using this to detect movement, once per second is fine. If you are trying to accumulate accelerations for inertial navigation, you will have to pick a sampling interval smaller than 1s. You've got GPS for navigation, so I assume you are just testing for movement.

3. Write data to SD. I am a little skeptical that all the prints can complete before the next GPS data barrage, but it's worth trying.

4. Write debug info. If you want some debug output, interleave a few, short messages, like Serial.println("open"). Few. Short. One character even. BTW, one-character strings like "," can be written ',' (a character literal), saving RAM.

"But how do I know when the GPS quiet time starts?" you ask.

Good question! Answer: When a few milliseconds have gone by without receiving any data.

This is from your first sketch:

   while (ss.available() > 0)
    if (gps.encode(ss.read()))
    { ...

I don't see this in your second sketch, so that's why no GPS data is received. Regardless, to detect the quiet time, you have to remember the last time you did see data with something like this:

   static unsigned long last_rx_time = 0;
   static bool waiting_for_quiet_time = true;

   // Handle GPS data
   while (ss.available() > 0) {
     if (gps.encode(ss.read())) {
       // set some variables, *NO PRINTS OR SD LOGGING*
       ...
     }
     last_rx_time = millis();
   }

   // See if the GPS device went quiet for a while.
   if (millis() - last_rx_time > 5UL) {

     if (waiting_for_quiet_time) {

       // Do these things just *once* per interval.
       // This flag gets reset when we start receiving data again.
       waiting_for_quiet_time = false;

       // Do all the other things!

       // Check Accel

       // Log to SD
         ...use the variables you set above, although most are probably still in the GPS object...

       // Serial.println( F("tick") );
     }

   } else {
     // We're getting data again, reset the flag.
     waiting_for_quiet_time = true;
   }

You could use the same technique to take accelerometer readings every 100ms:

   static unsigned long last_accel_reading = 0UL;
   if (millis() - last_accel_reading > 100UL) {
      last_accel_reading = millis();
      // take a reading
      ...
   }

Well, I would suggest starting with these suggestions. If you're still having trouble, you may have to move the GPS device to Serial, and use SoftwareSerial for debug output instead. In your final system, you may not even use the debug output.

BTW, which Arduino are you using?

Cheers, /dev

I just noticed that you are trying to log at 10Hz. I can't tell from the Venus spec if it outputs the NMEA sentences at the requested rate, or only Skytraq binary messages at rates higher than 1Hz.

I would suggest getting it working at 1Hz (the default), then try cranking the rate up. (Are you sure you need GPS at 10Hz?) At higher rates you may have to change how the quiet time is detected. The sentences are usually output in the same order, so watching for the last sentence in a batch would be better than waiting for a 5ms idle time.

For example, if the GPRMC is the last sentence in the batch, do all your other things after that sentence is gps.encoded'd. You will have to move the curSentenceType from the TinyGPS++ private section to the public section, then test the sentence type.

Cheers, /dev

Wow! Thanks for all the info! It is going to take me some time to digest all of this fully.

The distance sensor is actually to measure suspension travel of a certain component on a vehicle - so having a high refresh rate for this sensor is relatively critical. Likewise I need a high refresh rate for the accel.

I do suppose I could knock the GPS data down to 1hz, do you think that will help free up some RAM / CPU power to write the other data? If you think that would be an easy fix I can certainly try that out!

Thanks again to everyone!

I do suppose I could knock the GPS data down to 1hz, do you think that will help free up some RAM / CPU power to write the other data? If you think that would be an easy fix I can certainly try that out!

Yes, the GPS data is quite CPU intensive, especially when using SoftwareSerial. Depending on which Arduino you have (?), you might be able to use a real Serial port, not SoftwareSerial.

Normally, you have to send a special command to the GPS device to enable faster reporting rates. I do not see that in your code, so I believe the GPS is only running at 1Hz by default. That may be all you can do with the NMEA messages... without reading the Venus spec a little more carefully, I can't tell if you have to switch to the Skytraq binary protocol. TinyGPS is only for NMEA.

It does say that you have to increase the serial baudrate for higher reporting rates, and that requires a special binary command.

Start with 1Hz for everything, and make sure it's stable. Then bump the accel and retest. Then the distance... retest. Lastly, bump the GPS reporting rate. That will require reading the spec and maybe finding similar examples.

When you get to that point, I would probably point you toward a GPS library that I have written, called NeoGPS. I would not suggest using it at this point: you have too many other things to sort out before you tackle a more complicated library. You may not need it if you can get everything you need with NMEA. However, you will need it or something like it IFF you have to use the binary protocol. It wouldn't hurt you to read the NeoGPS document pages, as they enumerate some of the problems you have encountered, and will probably encounter. You might also read the NeoGPS announcement thread, which has some related discussion.

Cheers, /dev

/dev, thank you so much for your very detailed responses! I am using a ruggeduino currently (same as Uno).

Ill try slowing everything down and report back asap.

Thanks again!